Inter-core communication via uncore ram

ABSTRACT

A microprocessor includes a plurality of processing cores and an uncore random access memory (RAM) readable and writable by each of the plurality of processing cores. Each core of the plurality of processing cores comprises microcode run by the core that implements architectural instructions of an instruction set architecture of the microprocessor. The microcode is configured to both read and write the uncore RAM to accomplish inter-core communication between the plurality of processing cores.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application claims priority based on U.S. Provisional Application,Ser. No. 61/871,206, filed Aug. 28, 2013, and on U.S. ProvisionalApplication, Ser. No. 61/916,338, filed Dec. 16, 2013, each of which ishereby incorporated by reference in its entirety.

BACKGROUND

Multi-core microprocessors have proliferated, primarily due to theperformance advantages they offer. This has been made possible primarilyby the rapid reduction in semiconductor device geometry dimensionsresulting in increased transistor density. The presence of multiplecores in a microprocessor has created the need for the cores tocommunicate with one another in order to accomplish various featuressuch as power management, cache management, debugging, and configurationthat implicate more than one core.

Historically, architectural programs (e.g., operating system orapplication programs) running on multi-core processors have communicatedusing semaphores located in a system memory architecturally addressableby all the cores. This may suffice for many purposes, but may notprovide the speed, precision and/or system-level transparency needed forothers.

BRIEF SUMMARY

In one aspect the present invention provides a microprocessor. Themicroprocessor includes a plurality of processing cores and an uncorerandom access memory (RAM) readable and writable by each of theplurality of processing cores. Each core of the plurality of processingcores comprises microcode run by the core that implements architecturalinstructions of an instruction set architecture of the microprocessor.The microcode is configured to both read and write the uncore RAM toaccomplish inter-core communication between the plurality of processingcores.

In another aspect, the present invention provides a method to beperformed by a microprocessor having a plurality of processing cores andan uncore random access memory (RAM) readable and writable by each ofthe plurality of processing cores, wherein each core of the plurality ofprocessing cores comprises microcode run by the core that implementsarchitectural instructions of an instruction set architecture of themicroprocessor. The method includes reading, by the microcode, to theuncore RAM. The method also includes writing, by the microcode, from theuncore RAM. The reading and writing accomplish inter-core communicationbetween the plurality of processing cores.

In yet another aspect, the present invention provides a computer programproduct encoded in at least one non-transitory computer usable mediumfor use with a computing device, the computer program product comprisingcomputer usable program code embodied in said medium for specifying amicroprocessor. The computer usable program code includes first programcode for specifying a plurality of processing cores, and second programcode for specifying an uncore random access memory (RAM) readable andwritable by each of the plurality of processing cores. Each core of theplurality of processing cores comprises microcode run by the core thatimplements architectural instructions of an instruction set architectureof the microprocessor. The microcode is configured to both read andwrite the uncore RAM to accomplish inter-core communication between theplurality of processing cores.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a multi-core microprocessor.

FIG. 2 is a block diagram illustrating a control word, a status word,and a configuration word.

FIG. 3 is a flowchart illustrating operation of a control unit.

FIG. 4 is a block diagram illustrating an alternate embodiment of amicroprocessor.

FIG. 5 is a flowchart illustrating operation of a microprocessor to dumpdebug information.

FIG. 6 is a timing diagram illustrating an example of the operation of amicroprocessor according to the flowchart of FIG. 5.

FIG. 7 is a flowchart illustrating operation of a microprocessor toperform a trans-core cache control operation.

FIG. 8 is a timing diagram illustrating an example of the operation of amicroprocessor according to the flowchart of FIG. 7.

FIG. 9 is a flowchart illustrating operation of a microprocessor toenter a low power package C-state.

FIG. 10 is a timing diagram illustrating an example of the operation ofa microprocessor according to the flowchart of FIG. 9.

FIG. 11 is a flowchart illustrating operation of a microprocessor toenter a low power package C-state according to an alternate embodiment.

FIG. 12 is a timing diagram illustrating an example of the operation ofa microprocessor according to the flowchart of FIG. 11.

FIG. 13 is a timing diagram illustrating an alternate example of theoperation of a microprocessor according to the flowchart of FIG. 11.

FIG. 14 is a flowchart illustrating dynamic reconfiguration of amicroprocessor.

FIG. 15 is a flowchart illustrating dynamic reconfiguration of amicroprocessor according to an alternate embodiment.

FIG. 16 is a timing diagram illustrating an example of the operation ofa microprocessor according to the flowchart of FIG. 15.

FIG. 17 is a block diagram illustrating a hardware semaphore.

FIG. 18 is a flowchart illustrating operation of a hardware semaphorewhen read by a core.

FIG. 19 is a flowchart illustrating operation of a hardware semaphorewhen written by a core.

FIG. 20 is a flowchart illustrating operation of a microprocessor toemploy a hardware semaphore to perform an action that requires exclusiveownership of a resource.

FIG. 21 is a timing diagram illustrating an example of the operation ofa microprocessor according to the flowchart of FIG. 3 in which the coresissue non-sleeping sync requests.

FIG. 22 is a flowchart illustrating a process for configuring amicroprocessor.

FIG. 23 is a flowchart illustrating a process for configuring amicroprocessor according to an alternate embodiment.

FIG. 24 is a block diagram illustrating a multicore microprocessoraccording to an alternate embodiment.

FIG. 25 is a block diagram illustrating the structure of a microcodepatch.

FIG. 26 is a flowchart illustrating operation of the microprocessor ofFIG. 24 to propagate a microcode patch of FIG. 25 to multiple cores ofthe microprocessor.

FIG. 27 is a timing diagram illustrating an example of the operation ofa microprocessor according to the flowchart of FIG. 26.

FIG. 28 is a block diagram illustrating a multicore microprocessoraccording to an alternate embodiment.

FIG. 29 is a flowchart illustrating operation of the microprocessor ofFIG. 28 to propagate a microcode patch to multiple cores of themicroprocessor according to an alternate embodiment.

FIG. 30 is a flowchart illustrating operation of the microprocessor ofFIG. 24 to patch code for a service processor.

FIG. 31 is a block diagram illustrating a multicore microprocessoraccording to an alternate embodiment.

FIG. 32 is a flowchart illustrating operation of the microprocessor ofFIG. 31 to propagate an MTRR update to multiple cores of themicroprocessor.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Referring now to FIG. 1, a block diagram illustrating a multi-coremicroprocessor 100 is shown. The microprocessor 100 includes a pluralityof processing cores denoted 102A, 102B through 102N, which are referredto collectively as processing cores 102, or simply cores 102, and arereferred to individually as processing core 102, or simply core 102.Preferably, each core 102 includes one or more pipelines of functionalunits (not shown), including an instruction cache, instructiontranslation unit or instruction decoder that preferably includes amicrocode unit, register renaming unit, reservation stations, datacaches, execution units, memory subsystem and a retire unit including areorder buffer. Preferably, the cores 102 include a superscalar,out-of-order execution microarchitecture. In one embodiment, themicroprocessor 100 is an x86 architecture microprocessor, although otherembodiments are contemplated in which the microprocessor 100 conforms toanother instruction set architecture.

The microprocessor 100 also includes an uncore portion 103 coupled tothe cores 102 and that it is distinct from the cores 102. The uncore 103includes a control unit 104, fuses 114, a private random access memory(PRAM) 116, and a shared cache memory 119, for example, a level-2 (L2)and/or level-3 (L3) cache memory, shared by the cores 102. Each of thecores 102 is configured to read/write data from/to the uncore 103 via arespective address/data bus 126 that provides a non-architecturaladdress space (also referred to as private or micro-architecturaladdress space) to shared resources of the uncore 103. The PRAM 116 isprivate, or non-architectural, in the sense that it is not in thearchitectural user program address space of the microprocessor 100. Inone embodiment, the uncore 103 includes arbitration logic thatarbitrates requests by the cores 102 for access to uncore 103 resources.

Each of the fuses 114 is an electrical device that may be blown or notblown; when not blown, the fuse 114 has low impedance and readilyconducts electrical current; when blown, the fuse 114 has high impedanceand does not readily conduct electrical current. A sense circuit isassociated with each fuse 114 to evaluate the fuse 114, i.e., to sensewhether the fuse 114 conducts a high current or low voltage (not blown,e.g., logical zero, or clear) or a low current or high voltage (blown,e.g., logical one, or set). The fuse 114 may be blown during manufactureof the microprocessor 100 and, in some embodiments, an unblown fuse 114may be blown after manufacture of the microprocessor 100. Preferably,the blowing of a fuse 114 is irreversible. An example of a fuse 114 is apolysilicon fuse that may be blown by applying a sufficiently highvoltage across the device. Another example of a fuse 114 is anickel-chromium fuse that may be blown using a laser. Preferably, atpower up the sense circuit senses the fuse 114 and provides itsevaluation to a corresponding bit in a holding register of themicroprocessor 100. When the microprocessor 100 is released out ofreset, the cores 102 (e.g., microcode) read the holding registers todetermine the sensed fuse 114 values. In one embodiment, before themicroprocessor 100 is released out of reset, updated values may bescanned into the holding registers via a boundary scan input, forexample, such as a JTAG input, to essentially update the fuse 114values. This is particularly valuable for testing and/or debug purposes,such as in embodiments described below with respect to FIGS. 22 and 23.

Additionally, in one embodiment, the microprocessor 100 includes adifferent local Advanced Programmable Interrupt Controller (APIC) (notshown) associated with each core 102. In one embodiment, the local APICsconform architecturally to the description of a local APIC in the Intel64 and IA-32 Architectures Software Developer's Manual, Volume 3A, May2012, by the Intel Corporation, of Santa Clara, Calif., particularly insection 10.4. In particular, the local APIC includes an APIC ID registerthat includes an APIC ID and an APIC base register that includes abootstrap processor (BSP) flag, whose generation and uses are describedin more detail below, particularly with respect to embodiments relatedto FIGS. 14 through 16 and FIGS. 22 and 23.

The control unit 104 comprises hardware, software, or a combination ofhardware and software. The control unit 104 includes a hardwaresemaphore 118 (described in detail below with respect to FIGS. 17through 20), a status register 106, a configuration register 112, and arespective sync register 108 for each core 102. Preferably, each of theuncore 103 entities is addressable by each of the cores 102 at adistinct address within the non-architectural address space that enablesmicrocode to read and write it.

Each sync register 108 is writeable by its respective core 102. Thestatus register 106 is readable by each of the cores 102. Theconfiguration register 112 is readable and indirectly writeable (via thedisable core bit 236 of FIG. 2, as described below) by each of the cores102. The control unit 104 preferably includes interrupt logic (notshown) that generates a respective interrupt signal (INTR) 124 to eachcore 102, which the control unit 104 generates to interrupt therespective core 102. The interrupt sources in response to which thecontrol unit 104 generates an interrupt 124 to a core 102 may includeexternal interrupt sources, such as the x86 architecture INTR, SMI, NMIinterrupt sources; or bus events, such as the assertion or de-assertionof the x86 architecture-style bus signal STPCLK. Additionally, each core102 may send an inter-core interrupt 124 to each of the other cores 102by writing to the control unit 104. Preferably, the inter-coreinterrupts described herein, unless otherwise indicated, arenon-architectural inter-core interrupts requested by microcode of a core102 via a microinstruction, which are distinguished from conventionalarchitectural inter-core interrupts that system software requests via anarchitectural instruction. Finally, the control unit 104 may generate aninterrupt 124 to the cores 102 (a sync interrupt) when a synchronizationcondition, or sync condition, has occurred, as described below (e.g.,see FIG. 21 and block 334 of FIG. 3). The control unit 104 alsogenerates a respective core clock signal (CLOCK) 122 to each core 102,which the control unit 104 may selectively turn off and effectively putthe respective core 102 to sleep and turn on to wake the core 102 backup. The control unit 104 also generates a respective core power controlsignal (PWR) 128 to each core 102 that selectively controls whether ornot the respective core 102 is receiving power. Thus, the control unit104 may selectively turn off power to a core 102 via the respective PWRsignal 128 to put the core 102 into an even deeper sleep and turn powerback on to the core 102 as part of waking the core 102 up.

A core 102 may write to its respective sync register 108 with thesynchronization bit set (see S bit 222 of FIG. 2), which is referred toas a synchronization request, or sync request. As described in moredetail below, in one embodiment the sync request requests the controlunit 104 to put the core 102 to sleep and to awaken it when a synccondition occurs and/or when a specified wakeup event occurs. A synccondition occurs when all the enabled (see enabled bits 254 of FIG. 2)cores 102 of the microprocessor 100—or a specified subset of the enabledcores 102 (see FIG. 2 core set field 228)—have written the same synccondition (specified in a combination of the C bit 224, sync conditionor C-state field 226, and core set field 228 of FIG. 2, described inmore detail below with respect to the S bit 222) to their respectivesync register 108. In response to the occurrence of a sync condition,the control unit 104 simultaneously wakes up all the cores 102 that arewaiting on the sync condition, i.e., that have requested the synccondition. In an alternate embodiment described below, the cores 102 canrequest that only the last core 102 to write the sync request isawakened (see sel wake bit 214 of FIG. 2). In another embodiment, thesync request does not request to put the core 102 to sleep; instead, thesync request requests the control unit 104 to interrupt the cores 102when the sync condition occurs, as described in more detail below,particularly with respect to FIGS. 3 and 21.

Preferably, when the control unit 104 detects that a sync condition hasoccurred (due to the last core 102 writing the sync request to the syncregister 108), the control unit 104 puts the last core 102 to sleep,i.e., turns off the clock 122 to the last-writing core 102, and thensimultaneously awakes all the cores 102, i.e., turns on the clocks 122to all the cores 102. In this manner all the cores 102 are awakened,i.e., have their clocks 122 turned on, on precisely the same clockcycle. This may be particularly advantageous for certain operations,such as debugging (see for example embodiments of FIG. 5), in which itis beneficial for the cores 102 to wakeup on precisely the same clockcycle. In one embodiment, the uncore 103 includes a single phase-lockedloop (PLL) that produces the clock signals 122 provided to the cores102. In other embodiments, the microprocessor 100 includes multiple PLLsthat produces the clock signals 122 provided to the cores 102.

Control, Status and Configuration Words

Referring now to FIG. 2, a block diagram illustrating a control word202, a status word 242, and a configuration word 252 are shown. A core102 writes a value of the control word 202 to the sync register 108 ofthe control unit 104 of FIG. 1 to make an atomic request to sleep and/orto synchronize (sync) with all the other cores 102, or a specifiedsubset thereof, of the microprocessor 100. A core 102 reads a value ofthe status word 242 from the status register 106 of the control unit 104to determine status information described herein. A core 102 reads avalue of the configuration word 252 from the configuration register 112of the control unit 104 and uses the value as described below.

The control word 202 includes a wakeup events field 204, a sync controlfield 206, and a power gate (PG) bit 208. The sync control field 206includes various bits or subfields that control the sleeping of the core102 and/or the syncing of the core 102 with other cores 102. The synccontrol field 206 includes a sleep bit 212, a selective wakeup (selwake) bit 214, an S bit 222, a C bit 224, a sync condition or C-statefield 226, a core set field 228, a force sync bit 232, a selective synckill bit 234, and a core disable core bit 236. The status word 242includes a wakeup events field 244, a lowest common C-state field 246,and an error code field 248. The configuration word 252 includes oneenabled bit 254 for each core 102 of the microprocessor 100, a localcore number field 256, and a die number field 258.

The wakeup events field 204 of the control word 202 comprises aplurality of bits corresponding to different events. If the core 102sets a bit in the wakeup events field 204, the control unit 104 willwakeup (i.e., turn on the clock 122 to) the core 102 when the eventoccurs that corresponds to the bit. One wakeup event occurs when thecore 102 has synced with all the other cores specified in the core setfield 228. In one embodiment, the core set field 228 may specify all thecores 102 of the microprocessor 100; all the cores 102 that share acache memory (e.g., an L2 cache and/or L3 cache) with the instant core102; all the cores 102 on the same semiconductor die as the instant core102 (see FIG. 4 for an example of an embodiment that describes amulti-die, multi-core microprocessor 100); or all the cores 102 on theother semiconductor die as the instant core 102. A set of cores 102 thatshare a cache memory are referred to as a slice. Other examples ofwakeup events include, but are not limited to, an x86 INTR, SMI, NMI,assertion or de-assertion of STPCLK, and an inter-core interrupt. When acore 102 is awakened, it may read the wakeup events field 244 in thestatus word 242 to determine the active wakeup events.

If the core 102 sets the PG bit 208, the control unit 104 turns offpower to the core 102 (e.g., via the PWR signal 128) after it puts thecore 102 to sleep. When the control unit 104 subsequently restores powerto the core 102, the control unit 104 clears the PG bit 208. Use of thePG bit 208 is described in more detail below with respect to FIGS. 11through 13.

If the core 102 sets the sleep bit 212 or the sel wake bit 214, thecontrol unit 104 puts the core 102 to sleep after the core 102 writesthe sync register 108 using the wakeup events specified in the wakeupevents field 204. The sleep bit 212 and the sel wake bit 214 aremutually exclusive. The difference between them regards the action takenby the control unit 104 when a sync condition occurs. If a core 102 setsthe sleep bit 212, when a sync condition occurs the control unit 104will wake up all cores 102. In contrast, if a core 102 sets the sel wakebit 214, when the sync condition occurs the control unit 104 will wakeup only the last core 102 that wrote the sync condition to its syncregister 108.

If the core 102 sets neither the sleep bit 212 nor the sel wake bit 214,although the control unit 104 will not put the core 102 to sleep andwill therefore not wakeup the core 102 when a sync condition occurs, thecontrol unit 104 will nevertheless set the bit in the wakeup eventsfield 242 that indicates a sync condition is active, so the core 102 candetect the sync condition has occurred. Many of the wakeup events thatmay be specified in the wakeup events field 204 may also be interruptsources for which the control unit 104 can generate an interrupt to acore 102. However, the microcode of the core 102 may mask the interruptsources if desirable. If so, when the core 102 wakes up, the microcodemay read the status register 106 to determine whether a sync conditionoccurred or a wakeup event occurred or both.

If the core 102 sets the S bit 222, it requests the control unit 104 tosync on a sync condition. The sync condition is specified in somecombination of the C bit 224, sync condition or C-state field 226, andcore set field 228. If the C bit 224 is set, the C-state field 226specifies a C-state value; if the C bit 224 is clear, the sync conditionfield 226 specifies a non-C-state sync condition. Preferably, the valuesof the sync condition or C-state field 226 comprise a bounded set ofnon-negative integers. In one embodiment, the sync condition or C-statefield 226 is four bits. When the C bit 224 is clear, a sync conditionoccurs when all cores 102 in a specified core set 228 have written theirrespective sync register 108 with the S bit 222 set and with the samevalue of the sync condition field 226. In one embodiment, the synccondition field 226 values correspond to unique sync conditions, such asfor example, various sync conditions specified in the exemplaryembodiments described below. When the C bit 224 is set, a sync conditionoccurs when all cores 102 in a specified core set 228 have written theirrespective sync register 108 with the S bit 222 set regardless ofwhether they have written the same value of the C-state field 226. Inthis case, the control unit 104 posts the lowest written value of theC-state field 226 to the lowest common C-state field 246 of the statusregister 106, which may be read by a core 102 (e.g., by the master core102 at block 908 or by the last writing/selectively awakened core 102 atblock 1108). In one embodiment, if the core 102 specifies apredetermined value (e.g., all bits set) in the sync condition field226, this instructs the control unit 104 to match the instant core 102with any sync condition field 226 value specified by other cores 102.

If the core 102 sets the force sync bit 232, the control unit 104 forcesall pending sync requests to be immediately matched.

Normally, if any core 102 is awakened due to a wakeup event specified inthe wakeup events field 204, the control unit 104 kills all pending syncrequests (by clearing the S bit 222 in the sync register 108). However,if the core 102 sets the selective sync kill bit 234, the control unit104 will kill the pending sync request for only the core 102 that isawakened due to the (non-sync condition occurrence) wakeup event.

If two or more cores 102 request a sync on different sync conditions,the control unit 104 considers this a deadlock condition. Two or morecores 102 request a sync on different sync conditions if they writetheir respective sync register 108 with the S bit 222 set, the C bit 224clear and different values of the sync condition field 226. For example,if one core 102 writes to its sync register 108 with the S bit 222 setand the C bit 224 clear and a sync condition 226 value of 7 and anothercore 102 writes to its sync register 108 with the S bit 222 set and theC bit 224 clear and a sync condition 226 value of 9, then the controlunit 104 considers this a deadlock condition. Additionally, if one core102 writes to its sync register 108 with the C bit 224 clear and anothercore 102 writes to its sync register 108 with the C bit 224 set, thenthe control unit 104 considers this a deadlock condition. In response toa deadlock condition, the control unit 104 kills all pending syncrequests and wakes up all sleeping cores 102. The control unit 104 alsoposts values in the error code field 248 of the status register 106which the cores 102 may read to determine the cause of the deadlock andtake appropriate action. In one embodiment, the error code 248 indicatesthe sync condition written by each core 102, which enables each core todecide whether to proceed with its intended course of action or to deferto another core 102. For example, if one core 102 writes a synccondition to perform a power management operation (e.g., execute an x86MWAIT instruction) and another core 102 writes a sync condition toperform a cache management operation (e.g., x86 WBINVD instruction),then the core 102 that intended to perform the MWAIT defers to the core102 that is performing the WBINVD by cancelling the MWAIT, because theMWAIT is an optional operation, whereas the WBINVD is a mandatoryoperation. For another example, if one core 102 writes a sync conditionto perform a debug operation (e.g., to dump debug state) and anothercore 102 writes a sync condition to perform a cache management operation(e.g., WBINVD instruction), then the core 102 that intended to performthe WBINVD defers to the core 102 that is performing the debug dump bysaving the state of the WBINVD, waiting for the debug dump to occur, andthen restoring the state of the WBINVD and performing the WBINVD.

The die number field 258 is zero in a single-die embodiment. In amulti-die embodiment (e.g., FIG. 4), the die number field 258 indicateswhich die the core 102 reading the configuration register 112 resideson. For example, in a two-die embodiment, the dies are designated 0 and1 and the die number 258 has a value of either 0 or 1. In oneembodiment, fuses 114 are selectively blown to designate a die as 0 or1, for example.

The local core number field 256 indicates the core number, local to itsdie, of the core 102 that is currently reading the configurationregister 112. Preferably, although there is a single configurationregister 112 shared by all the cores 102, the control unit 104 knowswhich core 102 is reading the configuration register 112 and providesthe correct value in the local core number field 256 based on thereader. This enables microcode of the core 102 to know its local corenumber among the other cores 102 located on the same die. In oneembodiment, a multiplexer in the uncore 103 portion of themicroprocessor 100 selects the appropriate value that is returned in thelocal core number field 256 of the configuration word 252 depending uponthe core 102 reading the configuration register 112. In one embodiment,selectively blown fuses 114 operate in conjunction with the multiplexerto return the local core number field 256 value. Preferably, the localcore number field 256 value is fixed independent of which cores 102 onthe die are enabled, as indicated by the enabled bits 254 describedbelow. That is, even if one or more cores 102 on the die are disabled,the local core number field 256 values remain fixed. Additionally, themicrocode of a core 102 computes the global core number of the core 102,which is a configuration-related value, whose use is described in moredetail below. The global core number indicates the core number of thecore 102 global to the microprocessor 100. The core 102 computes itsglobal core number by using the die number field 258 value. For example,in an embodiment in which the microprocessor 100 includes eight cores102 evenly divided on two dies having die numbers 0 and 1, on each diethe local core number field 256 returns a value of either 0, 1, 2 or 3;the cores 102 on die number 1 add 4 to the value returned in the localcore number field 256 to compute their global core number.

Each core 102 of the microprocessor 100 has a corresponding enabled bit254 of the configuration word 252 that indicates whether the core 102 isenabled or disabled. In FIG. 2, the enabled bits 254 are individuallydenoted enabled bit 254-x, where x is the global core number of thecorresponding core 102. The example of FIG. 2 assumes eight cores 102 onthe microprocessor 100. In the example of FIGS. 2 and 4, enabled bit254-0 indicates whether the core 102 having global core number 0 (e.g.,core A) is enabled, enabled bit 254-1 indicates whether the core 102having global core number 1 (e.g., core B) is enabled, enabled bit 254-2indicates whether the core 102 having global core number 2 (e.g., coreC) is enabled, and so forth. Thus, by knowing its global core number,microcode of a core 102 can determine from the configuration word 252which cores 102 of the microprocessor 100 are disabled and which areenabled. Preferably, an enabled bit 254 is set if the core 102 isenabled and is clear if the core 102 is disabled. When themicroprocessor 100 is reset, hardware automatically populates theenabled bits 254. Preferably, the hardware populates the enabled bits254 based on fuses 114 selectively blown when the microprocessor 100 ismanufactured that indicate whether a given core 102 is enabled ordisabled. For example, if a given core 102 is tested and found to befaulty, a fuse 114 may be blown to clear the enabled bit 254 of the core102. In one embodiment, a fuse 114 blown to indicate a core 102 isdisabled also prevents clock signals from being provided to the disabledcore 102. Each core 102 can write the disable core bit 236 in its syncregister 108 to clear its enabled bit 254, as described in more detailbelow with respect to FIGS. 14 through 16. Preferably, clearing theenabled bit 254 does not prevent the core 102 from executinginstructions, but simply updates the configuration register 112, and thecore 102 must set a different bit (not shown) to prevent itself fromexecuting instructions, e.g., to have its power removed and/or turn offits clock signals. For a multi-die configuration microprocessor 100(e.g., FIG. 4), the configuration register 112 includes an enabled bit254 for all cores 102 of the microprocessor 100, i.e., not just thecores 102 of the local die but also the cores 102 of the remote die.Preferably, in the case of a multi-die configuration microprocessor 100,when a core 102 writes to its sync register 108, the sync register 108value is propagated to the core's 102 corresponding shadow sync register108 on the other die (see FIG. 4), which, if the disable core bit 236 isset, causes an update to the remote die configuration register 112 suchthat both the local and remote die configuration registers 112 have thesame value.

In one embodiment, the configuration register 112 cannot be writtendirectly by a core 102; however, a write by a core 102 to theconfiguration register 112 causes the local enabled bit 254 values to bepropagated to the configuration register 112 of the other die in amulti-die microprocessor 100 configuration, as described with respect toblock 1406 of FIG. 14, for example.

Control Unit

Referring now to FIG. 3, a flowchart illustrating operation of thecontrol unit 104 is shown. Flow begins at block 302.

At block 302, a core 102 writes a sync request, i.e., writes to its syncregister 108 a control word 202, which is received the control unit 104.In the case of a multi-die configuration microprocessor 100 (e.g., seeFIG. 4), when a shadow register 108 of the control unit 104 receives apropagated sync register 108 value from the other die 406, the controlunit 104 operates effectively according to FIG. 3, i.e., as if thecontrol unit 104 received a sync request from one of its local cores 102(at block 302), except the control unit 104 only puts to sleep (e.g., atblock 314) or wakes up (at blocks 306 or 328 or 336) or interrupts (atblock 334) or blocks wakeup events for (at block 326) cores 102 on itslocal die 406 and only populates its local status register 106 (at block318). Flow proceeds to block 304.

At block 304, the control unit 104 examines the sync condition specifiedat block 302 to determine if a deadlock condition has occurred, asdescribed above with respect to FIG. 2. If so, flow proceeds to block306; otherwise, flow proceeds to decision block 312.

At block 305, the control unit 104 detects the occurrence of a wakeupevent specified in the wakeup events field 204 of one of the syncregisters 108 (other than a sync condition occurrence, which is detectedat block 316). As described below with respect to block 326, the controlunit 104 may automatically block the wakeup events. The control unit 104may detect the wakeup event occurrence as an event asynchronous to thewriting of a sync request at block 302. Flow proceeds also from block305 to block 306.

At block 306, the control unit 104 populates the status register 106,kills pending sync requests, and wakes up any sleeping cores 102. Asdescribed above, waking up a sleeping core 102 may include restoring itspower. The cores 102 may then read the status register 106, inparticular the error code 248, to determine the cause of the deadlockand handle it based on the relative priorities of the conflicting syncrequests, as described above. Additionally, the control unit 104 killsall pending sync requests (i.e., clears the S bit 222 in the syncregister 108 of each of the cores 102), unless block 306 was reachedfrom block 305 and the selective sync kill bit 234 was set, in whichcase the control unit 104 will kill the pending sync request of only thecore 102 being awakened by the wakeup event. If block 306 was reachedfrom block 305, the cores 102 may read the wakeup events 244 field todetermine the wakeup event that occurred. Additionally, if the wakeupevent was an unmasked interrupt source, the control unit 104 willgenerate an interrupt request via the INTR signal 124 to the core 102.Flow ends at block 306.

At decision block 312, the control unit 104 determines whether the sleep212 or sel wake bit 214 is set. If so, flow proceeds to block 314;otherwise, flow proceeds to decision block 316.

At block 314, the control unit 104 puts the core 102 to sleep. Asdescribed above, putting a core 102 to sleep may include removing itspower. In one embodiment, as an optimization, even if the PG bit 208 isset, the control unit 104 does not remove power from the core 102 atblock 314 if this is the last writing core 102 (i.e., will cause thesync condition to occur) and the sel wake bit 214 is set since thecontrol unit 104 will be immediately waking the last writing core 102back up at block 328. In one embodiment, the control unit 104 comprisessynchronization logic and sleep logic, which are separate from, but incommunication with, one other; furthermore, the sync logic and sleeplogic each comprise a portion of the sync register 108. Advantageously,the write to the sync logic portion of the sync register 108 and thewrite to the sleep logic portion of the sync register 108 are atomic.That is, if one occurs, they are both guaranteed to occur. Preferably,the core 102 pipeline stalls, not allowing any more writes to occur,until it is guaranteed that the writes to both portions of the syncregister 108 have occurred. An advantage of writing a sync request andimmediately sleeping is that it does not require the core 102 (e.g.,microcode) to continuously loop to determine whether the sync conditionhas occurred. This is advantageous because it saves power and does notconsume other resources, such as bus and/or memory bandwidth. It isnoted that the core 102 may write to the sync register 108 with the Sbit 222 clear and the sleep bit 212 set, referred to herein as a sleeprequest, in order to sleep but without requesting a sync with othercores 102 (e.g., at blocks 924 and 1124); in this case the control unit104 wakes up the core 102 (e.g., at block 306) if an unmasked wakeupevent specified in the wakeup events field 204 occurs (e.g., at block305), but does not look for a sync condition occurrence for this core102 (e.g., at block 316). Flow proceeds to decision block 316.

At decision block 316, the control unit 104 determines whether a synccondition occurred. If so, flow proceeds to block 318. As describedabove, a sync condition can occur only if the S bit 222 is set. In oneembodiment, the control unit 104 uses the enabled bits 254 of FIG. 2that indicate which cores 102 in the microprocessor 100 are enabled andwhich cores 102 are disabled. The control unit 104 only looks for thecores 102 that are enabled to determine whether a sync condition hasoccurred. A core 102 may be disabled because it was tested and founddefective at manufacturing time; consequently, a fuse was blown to keepthe core 102 from operating and to indicate the core 102 is disabled. Acore 102 may be disabled because software requested the core 102 bedisabled (e.g., see FIG. 15). For example, at a user request, BIOSwrites to a model specific register (MSR) to request the core 102 bedisabled, and in response the core 102 disables itself (e.g., via thedisable core bit 236) and notifies the other cores 102 to read theconfiguration register 112 by which the other cores 102 determine thecore 102 is disabled. A core 102 may also be disabled via a microcodepatch (e.g., see FIG. 14), which may be made by blowing fuses 114 and/orloaded from system memory, such as a FLASH memory. In addition todetermining whether a sync condition occurred, the control unit 104examines the force sync bit 232. If set, flow also proceeds to block318. If the force sync bit 232 is clear and a sync condition has notoccurred, flow ends at block 316.

At block 318, the control unit 104 populates the status register 106.Specifically, if the occurring sync condition was that all cores 102requested a C-state sync, the control unit 104 populates the lowestcommon C-state field 246 as described above. Flow proceeds to decisionblock 322.

At decision block 322, the control unit 104 examines the sel wake bit214. If the bit is set, flow proceeds to block 326; otherwise, flowproceeds to decision block 332.

At block 326, the control unit 104 blocks all wakeup events for allother cores 102 except the instant core 102, which was last core 102 towrite the sync request to its sync register 108 at block 302 andtherefore to cause the sync condition to occur. In one embodiment, logicof the control unit 104 simply Boolean ANDs the wakeup conditions with asignal that is false if it is desired to block the wakeup events andotherwise is true. A use for blocking off all the wakeup events for allthe other cores is described in more detail below, particularly withrespect to FIGS. 11 through 13. Flow proceeds to block 328.

At block 328, the control unit 104 wakes up only the instant core 102,but does not wakeup the other cores that requested the sync.Additionally, the control unit 104 kills the pending sync request forthe instant core 102 by clearing its S bit 222, but does not kill thepending sync requests for the other cores 102, i.e., leaves the S bit222 set for the other cores 102. Consequently and advantageously, if andwhen the instant core 102 writes another sync request after it isawakened, it will again cause the sync condition to occur (assuming thepending sync requests of the other cores 102 have not been killed), anexample of which is described below with respect to FIGS. 12 and 13.Flow ends at block 328.

At decision block 332, the control unit 104 examines the sleep bit 212.If the bit is set, flow proceeds to block 336; otherwise, flow proceedsto block 334.

At block 334, the control unit 104 sends an interrupt (a sync interrupt)to all the cores 102. The timing diagram of FIG. 21 illustrates anexample of a non-sleeping sync request. Each core 102 may read thewakeup events field 244 and detect that a sync condition occurrence wasthe cause of the interrupt. Flow has proceeded to block 334 in the casewhere the cores 102 elected not to go to sleep when they wrote theirsync requests. Although this case does not enable them to enjoy the samebenefit (i.e., simultaneous wakeup) of the case where they sleep, it hasthe potential advantage of allowing the cores 102 to continue processinginstructions while waiting for the last core 102 to write its syncrequest in situations where simultaneous wakeup is not needed. Flow endsat block 334.

At block 336, the control unit 104 simultaneously wakes up all the cores102. In one embodiment, the control unit 104 turns on the clocks 122 toall the cores 102 on precisely the same clock cycle. In anotherembodiment, the control unit 104 turns on the clocks 122 to all thecores 102 in a staggered fashion. That is, the control unit 104introduces a delay of a predetermined number of clock cycles (e.g., onthe order of ten or a hundred clocks) in between turning on the clock122 to each core 102. However, the staggered turning on of the clocks122 is considered simultaneous in the present disclosure. It may beadvantageous to stagger turning on the clocks 122 in order to reduce thelikelihood of a power consumption spike when all the cores 102 wake up.In yet another embodiment, in order to reduce the power consumptionspike likelihood, the control unit 104 turns on the clock signals 122 toall the cores 102 on the same clock cycle, but does so in a stuttering,or throttled, fashion by initially providing the clock signals 122 at areduced frequency and ramping up the frequency to the target frequency.In one embodiment, the sync requests are issued as a result of theexecution of an instruction of microcode of the core 102, and themicrocode is designed such that, for at least some of the sync conditionvalues, the location in the microcode that specifies the sync conditionvalue is unique. For example, only one place in the microcode includes async x request, only one place in the microcode includes a sync yrequest, and so forth. In these cases, the simultaneous wakeup isadvantageous because all cores 102 are waking up in the exact sameplace, which enables the microcode designer to design more efficient andbug-free code. Furthermore, the simultaneous wakeup may be particularlyadvantageous for debugging purposes when attempting to recreate and fixbugs that only appear due to the interaction of multiple cores but thatdo not appear when a single core is running FIGS. 5 and 6 depict such anexample. Additionally, the control unit 104 kills all pending syncrequests (i.e., clears the S bit 222 in the sync register 108 of each ofthe cores 102). Flow ends at block 336.

An advantage of embodiments described herein is that they maysignificantly reduce the amount of microcode in a microprocessorbecause, rather than looping or performing other checks to synchronizeoperations between multiple cores, the microcode in each core can simplywrite the sync request, go to sleep, and know that when it wakes up allthe cores are in the same place in microcode. Microcode uses of the syncrequest mechanism will be described below.

Multi-Die Microprocessor

Referring now to FIG. 4, a block diagram illustrating an alternateembodiment of a microprocessor 100 is shown. The microprocessor 100 ofFIG. 4 is similar in many respects to the microprocessor 100 of FIG. 1in that it is a multi-core processor and the cores 102 are similar.However, the embodiment of FIG. 4 is a multi-die configuration. That is,the microprocessor 100 comprises multiple semiconductor dies 406 mountedwithin a common package and in communication with one another via aninter-die bus 404. The embodiment of FIG. 4 includes two dies 406,denoted die A 406A and die B 406B coupled by the inter-die bus 404.Furthermore, each die 406 comprises an inter-die bus unit 402 thatinterfaces its respective die 406 to the inter-die bus 404. Stillfurther, each die 406 includes its own uncore 103 control unit 104coupled to its respective cores 102 and inter-die bus unit 402. In theembodiment of FIG. 4, die A 406A includes four cores 102—core A 102A,core B 102B, core C 102C and core D 102D that are coupled to a controlunit A 102A, which is coupled to an inter-die bus unit A 402A;similarly, die B 406B includes four cores 102—core E 102E, core F 102F,core G 102G and core H 102H that are coupled to a control unit B 102B,which is coupled to an inter-die bus unit B 402B. Finally, each of thecontrol units 104 includes not only a sync register 108 for each of thecores 102 on the die 406 that comprises it, but also includes a syncregister 108 for each of the cores 102 on the other die 406, which arereferred to as shadow registers in FIG. 4. Thus, each of the controlunits 104 of the embodiment of FIG. 4 includes eight sync register 108,denoted 108A, 108B, 108C, 108D, 108E, 108F, 108G and 108H. In controlunit A 104A, sync registers 108E, 108F, 108G and 108H are the shadowregisters, whereas in control unit B 104B, sync registers 108A, 108B,108C and 108D are the shadow registers.

When a core 102 writes a value to its sync register 108, the controlunit 104 on the core's 102 die 406 writes the value, via the inter-diebus units 402 and inter-die bus 404, to the corresponding shadowregister 108 on the other die 406. Furthermore, if the disable core bit236 is set in the value propagated to the shadow sync register 108, thecontrol unit 104 also updates the corresponding enabled bit 254 in theconfiguration register 112. In this manner, a sync conditionoccurrence—including a trans-die sync condition occurrence—may bedetected even in situations in which the microprocessor 100 coreconfiguration is dynamically changing (e.g., FIG. 14 through 16). In oneembodiment, the inter-die bus 404 is a relatively low-speed bus, and thepropagation may take on the order of 100 core clock cycles that is apredetermined number, and each of the control units 104 comprises astate machine that takes a predetermined number of clocks to detect thesync condition occurrence and turn on the clocks to all the cores 102 ofits respective die 406. Preferably, the control unit 104 on the localdie 406 (i.e., the die 406 comprising the core 102 that wrote) isconfigured to delay updating the local sync register 108 until apredetermined number of clocks (e.g., the sum of the number ofpropagation clocks and the number of state machine sync conditionoccurrence detection clocks) after initiating the write of the value tothe other die 406 (e.g., being granted the inter-die bus 404). In thismanner, the control units 104 on both dies simultaneously detect theoccurrence of a sync condition and turn on the clocks to all cores 102on both dies 406 at the same time. This may be particularly advantageousfor debugging purposes when attempting to recreate and fix bugs thatonly appear due to the interaction of multiple cores but that do notappear when a single core is running FIGS. 5 and 6 describe embodimentsthat may take advantage of this feature.

Debug Operations

The cores 102 of the microprocessor 100 are configured to performindividual debug operations, such as breakpoints on instructionexecutions and data accesses. Furthermore, the microprocessor 100 isconfigured to perform debug operations that are trans-core, i.e., thatimplicate more than one core 102 of the microprocessor 100.

Referring now to FIG. 5, a flowchart illustrating operation of themicroprocessor 100 to dump debug information is shown. The operation isdescribed from the perspective of a single core, but each of the cores102 of the microprocessor 100 operates according to the description tocollectively dump the state of the microprocessor 100. Morespecifically, FIG. 5 describes the operation of one core that receivesthe request to dump the debug information, whose flow begins at block502, and the operation of the other cores 102, whose flow begins atblock 532.

At block 502, one of the cores 102 receives a request to dump debuginformation. Preferably, the debug information includes the state of thecore 102 or a subset thereof. Preferably, the debug information isdumped to system memory or to an external bus that may be monitored bydebug equipment, such as a logic analyzer. In response to the request,the core 102 sends a debug dump message to the other cores 102 and sendsthem an inter-core interrupt. Preferably, the core 102 traps tomicrocode in response to the request to dump the debug information (atblock 502) or in response to the interrupt (at block 532) and remains inmicrocode, during which time interrupts are disabled (i.e., themicrocode does not allow itself to be interrupted), until block 528. Inone embodiment, the core 102 only takes interrupts when it is asleep andon architectural instruction boundaries. In one embodiment, variousinter-core messages described herein (such as the message sent at block502 and other messages, such as at blocks 702, 1502, 2606 and 3206) aresent and received via the sync condition or C-state field 226 of thecontrol word 202 of the sync registers 108. In other embodiments, theinter-core messages are sent and received via the uncore PRAM 116. Flowproceeds from block 502 to block 504.

At block 532, one of the other cores 102 (i.e., a core 102 other thanthe core 102 that received the debug dump request at block 502) getsinterrupted and receives the debug dump message as a result of theinter-core interrupt and message sent at block 502. As described above,although flow at block 532 is described from the perspective of a singlecore 102, each of the other cores 102 (i.e., not the core 102 at block502) gets interrupted and receives the message at block 532 and performsthe steps at blocks 504 through 528. Flow proceeds from block 532 toblock 504.

At block 504, the core 102 writes a sync request to its sync register108 with a sync condition value of 1 (denoted sync 1 in FIG. 5). As aresult, the control unit 104 puts the core 102 to sleep. Flow proceedsto block 506.

At block 506, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 1. Flow proceeds to block 508.

At block 508, the core 102 dumps its state to memory. Flow proceeds toblock 514.

At block 514, the core 102 writes a sync 2, which results in the controlunit 104 putting the core 102 to sleep. Flow proceeds to block 516.

At block 516, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 2. Flow proceeds to block 518.

At block 518, the core 102 saves the address of the memory location towhich the debug information was dumped at block 508 and sets a flag,both of which persist through a reset, and then resets itself The core's102 reset microcode detects the flag and reloads its state from thesaved memory location. Flow proceeds to block 524.

At block 524, the core 102 writes a sync 3, which results in the controlunit 104 putting the core 102 to sleep. Flow proceeds to block 526.

At block 526, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 3. Flow proceeds to block 528.

At block 528, the core 102 comes out of reset and begins fetchingarchitectural (e.g., x86) instructions based on the state that wasreloaded at block 518. Flow ends at block 528.

Referring now to FIG. 6, a timing diagram illustrating an example of theoperation of the microprocessor 100 according to the flowchart of FIG. 5is shown. In the example, a configuration of a microprocessor 100 withthree cores 102, denoted core 0, core 1 and core 2, is shown; however,it should be understood that in other embodiments the microprocessor 100may include different numbers of cores 102. In the timing diagram, thetiming of events proceeds downward.

Core 0 receives a debug dump request and in response sends a debug dumpmessage and interrupt to core 1 and core 2 (per block 502). Core 0 thenwrites a sync 1 and is put to sleep (per block 504).

Each of core 1 and core 2 eventually are interrupted from their currenttasks and read the message (per block 532). In response, each of core 1and core 2 writes a sync 1 and is put to sleep (per block 504). Asshown, the time at which each of the cores writes the sync 1 may vary,for example due to the latency of the instruction that is executing whenthe interrupt is asserted.

When all the cores have written the sync 1, the control unit 104 wakesthem all up simultaneously (per block 506). Each core then dumps itsstate to memory (per block 508) and writes a sync 2 and is put to sleep(per block 514). The amount of time required to dump the state may vary;consequently, the time at which each of the cores writes the sync 2 mayvary, as shown.

When all the cores have written the sync 2, the control unit 104 wakesthem all up simultaneously (per block 516). Each core then resets itselfand reloads its state from memory (per block 518) and writes a sync 3and is put to sleep (per block 524). As shown, the amount of timerequired to reset and reload the state may vary; consequently, the timeat which each of the cores writes the sync 3 may vary.

When all the cores have written the sync 3, the control unit 104 wakesthem all up simultaneously (per block 526). Each core then beginsfetching architectural instructions at the point where it wasinterrupted (per block 528).

A conventional solution to synchronizing operations between multipleprocessors is to employ software semaphores. However, a disadvantage ofthe conventional solution is that they do not provide clock-levelsynchronization. An advantage of the embodiments described herein isthat the control unit 104 can turn on the clocks 122 to all of the cores102 simultaneously.

In the manner described above, an engineer debugging the microprocessor100 may configure one of the cores 102 to periodically generatecheckpoints at which it generates the debug dump requests, for exampleafter a predetermined number of instructions have been retired. Whilethe microprocessor 100 is running, the engineer captures all activity onthe external bus of the microprocessor 100 in a log. The portion of thelog near the time the bug is suspected to have occurred may then beprovided to a software simulator that simulates the microprocessor 100to aid the engineer in debugging. The simulator simulates the executionof instructions by each core 102 and simulates the transactions on theexternal microprocessor 100 bus using the log information. In oneembodiment, the simulators for all the cores 102 are startedsimultaneously from a reset point. Therefore, it is highly desirablethat all the cores 102 of the microprocessor 100 actually come out ofreset (e.g., after the sync 2) at the same time. Furthermore, by waitingto dump its state until all the other cores 102 have stopped theircurrent task (e.g., after the sync 1), the dumping of the state by onecore 102 does not interfere with the execution by the other cores 102 ofcode and/or hardware that is being debugged (e.g., shared memory bus orcache interaction), which may increase the likelihood of being able toreproduce the bug and determine its cause. Similarly, waiting to beginfetching architectural instructions until all the cores 102 havefinished reloading their state (e.g., after the sync 3), reloading ofthe state by one core 102 does not interfere with the execution by theother cores 102 of code and/or hardware that is being debugged, whichmay increase the likelihood of being able to reproduce the bug anddetermine its cause. These advantages may provide benefits over priormethods such as described in U.S. Pat. No. 8,370,684, which is herebyincorporated by reference in its entirety for all purposes, which didnot enjoy the advantage of cores being able to make sync requests.

Cache Control Operations

The cores 102 of the microprocessor 100 are configured to performindividual cache control operations, such as on local cache memories,i.e., caches that are not shared by two or more cores 102. Furthermore,the microprocessor 100 is configured to perform cache control operationsthat are trans-core, i.e., that implicate more than one core 102 of themicroprocessor 100, e.g., because they implicate a shared cache 119.

Referring now to FIG. 7, a flowchart illustrating operation of themicroprocessor 100 to perform a trans-core cache control operation isshown. The embodiment of FIG. 7 describes how the microprocessor 100performs an x86 architecture write-back-and-invalidate cache (WBINVD)instruction. A WBINVD instruction instructs the core 102 executing theinstruction to write back all modified cache lines in the cache memoriesof the microprocessor 100 to system memory and to invalidate, or flush,the cache memories. The WBINVD instruction also instructs the core 102to issue special bus cycles to direct any cache memories external to themicroprocessor 100 to write back their modified data and invalidatethemselves. The operation is described from the perspective of a singlecore, but each of the cores 102 of the microprocessor 100 operatesaccording to the description to collectively write back modified cachelines and invalidate the cache memories of the microprocessor 100. Morespecifically, FIG. 7 describes the operation of one core that encountersthe WBINVD instruction, whose flow begins at block 702, and theoperation of the other cores 102, whose flow begins at block 752.

At block 702, one of the cores 102 encounters a WBINVD instruction. Inresponse, the core 102 sends a WBINVD instruction message to the othercores 102 and sends them an inter-core interrupt. Preferably, the core102 traps to microcode in response to the WBINVD instruction (at block702) or in response to the interrupt (at block 752) and remains inmicrocode, during which time interrupts are disabled (i.e., themicrocode does not allow itself to be interrupted), until block 748/749.Flow proceeds from block 702 to block 704.

At block 752, one of the other cores 102 (i.e., a core 102 other thanthe core 102 that encountered the WBINVD instruction at block 702) getsinterrupted and receives the WBINVD instruction message as a result ofthe inter-core interrupt sent at block 702. As described above, althoughflow at block 752 is described from the perspective of a single core102, each of the other cores 102 (i.e., not the core 102 at block 702)gets interrupted and receives the message at block 752 and performs thesteps at blocks 704 through 749. Flow proceeds from block 752 to block704.

At block 704, the core 102 writes a sync request to its sync register108 with a sync condition value of 4 (denoted sync 4 in FIG. 7). As aresult, the control unit 104 puts the core 102 to sleep. Flow proceedsto block 706.

At block 706, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 4. Flow proceeds to block 708.

At block 708, the core 102 writes back and invalidates its local cachememories, e.g., level-1 (L1) cache memories that are not shared by thecore 102 with other cores 102. Flow proceeds to block 714.

At block 714, the core 102 writes a sync 5, which results in the controlunit 104 putting the core 102 to sleep. Flow proceeds to block 716.

At block 716, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 5. Flow proceeds to decision block717.

At decision block 717, the core 102 determines whether it was the core102 that encountered the WBINVD instruction at block 702 (as opposed toa core 102 that received the WBINVD instruction message at block 752).If so, flow proceeds to block 718; otherwise, flow proceeds to block724.

At block 718, the core 102 writes back and invalidates the shared cache119. In one embodiment, the microprocessor 100 comprises slices in whichmultiple, but not all, cores 102 of the microprocessor 100 share a cachememory, as described above. In such embodiments, intermediate operations(not shown) similar to blocks 717 through 726 are performed in which oneof the cores 102 in the slice writes back and invalidates the sharedcache memory while the other core(s) of the slice go back to sleepsimilar to block 724 to wait until the slice cache memory isinvalidated. Flow proceeds to block 724.

At block 724, the core 102 writes a sync 6, which results in the controlunit 104 putting the core 102 to sleep. Flow proceeds to block 726.

At block 726, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 6. Flow proceeds to decision block727.

At decision block 727, the core 102 determines whether it was the core102 that encountered the WBINVD instruction at block 702 (as opposed toa core 102 that received the WBINVD instruction message at block 752).If so, flow proceeds to block 728; otherwise, flow proceeds to block744.

At block 728, the core 102 issues the special bus cycles to causeexternal caches to be written back and invalidated. Flow proceeds toblock 744.

At block 744, the core 102 writes a sync 13, which results in thecontrol unit 104 putting the core 102 to sleep. Flow proceeds to block746.

At block 746, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 13. Flow proceeds to decision block747.

At decision block 747, the core 102 determines whether it was the core102 that encountered the WBINVD instruction at block 702 (as opposed toa core 102 that received the WBINVD instruction message at block 752).If so, flow proceeds to block 748; otherwise, flow proceeds to block749.

At block 748, the core 102 completes the WBINVD instruction, whichincludes retiring the WBINVD instruction and may include relinquishingownership of a hardware semaphore (see FIG. 20). Flow ends at block 748.

At block 749, the core 102 resumes the task it was performing before itwas interrupted at block 752. Flow ends at block 749.

Referring now to FIG. 8, a timing diagram illustrating an example of theoperation of the microprocessor 100 according to the flowchart of FIG. 7is shown. In the example, a configuration of a microprocessor 100 withthree cores 102, denoted core 0, core 1 and core 2, is shown; however,it should be understood that in other embodiments the microprocessor 100may include different numbers of cores 102.

Core 0 encounters a WBINVD instruction and in response sends a WBINVDinstruction message and interrupt to core 1 and core 2 (per block 702).Core 0 then writes a sync 4 and is put to sleep (per block 704).

Each of core 1 and core 2 eventually are interrupted from their currenttasks and read the message (per block 752). In response, each of core 1and core 2 writes a sync 4 and is put to sleep (per block 704). Asshown, the time at which each of the cores writes the sync 4 may vary.

When all the cores have written the sync 4, the control unit 104 wakesthem all up simultaneously (per block 706). Each core then writes backand invalidates it unique cache memories (per block 708) and writes async 5 and is put to sleep (per block 714). The amount of time requiredto write back and invalidate the cache may vary; consequently, the timeat which each of the cores writes the sync 5 may vary, as shown.

When all the cores have written the sync 5, the control unit 104 wakesthem all up simultaneously (per block 716). Only the core thatencountered the WBINVD instruction writes back and invalidates theshared cache 119 (per block 718) and all of the cores write a sync 6 andare put to sleep (per block 724). Since only one core writes back andinvalidates the shared cache 119, the time at which each of the coreswrites the sync 6 may vary, as shown.

When all the cores have written the sync 6, the control unit 104 wakesthem all up simultaneously (per block 726). Only the core thatencountered the WBINVD instruction completes the WBINVD instruction (perblock 748) and all of the other cores resume their pre-interruptprocessing (per block 749).

It should be understood that although embodiments have been described inwhich the cache control instruction is an x86 WBINVD instruction, otherembodiments are contemplated in which sync requests are employed toperform other cache control instructions. For example, themicroprocessor 100 may perform similar actions to perform an x86 INVDinstruction without writing back the cache data (at blocks 708 and 718)and simply invalidating the caches. For another example, the cachecontrol instruction may be from a different instruction set architecturethan the x86 architecture.

Power Management Operations

The cores 102 of the microprocessor 100 are configured to performindividual power reduction actions, such as, but not limited to, ceasingto execute instructions, requesting the control unit 104 to stop clocksignals to the core 102, requesting the control unit 104 to remove powerfrom the core 102, writing back and invalidating local (i.e.,non-shared) cache memories of the core 102 and saving the state of thecore 102 to an external memory such as the PRAM 116. When a core 102 hasperformed one or more core-specific power reduction actions it hasentered a “core” C-state (also referred to as a core idle state or coresleep state). In one embodiment, the C-state values may correspondroughly to the well-known Advanced Configuration and Power Interface(ACPI) Specification Processor states, but may include finergranularity. Typically, a core 102 will enter a core C-state in responseto a request from the operating system to do so. For example, the x86architecture monitor wait (MWAIT) instruction is a power managementinstruction that provides a hint, namely a target C-state, to the core102 executing the instruction to allow the microprocessor 100 to enteran optimized state, such as a lower power consuming state. In the caseof an MWAIT instruction, the target C-states are proprietary rather thanbeing ACPI C-states. Core C-state 0 (C0) corresponds to the runningstate of the core 102 and increasingly larger values of the C-statecorrespond to increasingly less active or responsive states (such as theC1, C2, C3, etc. states). A progressively less responsive or activestate refers to a configuration or operating state that saves morepower, relative to a more active or responsive state, or is somehowrelatively less responsive (e.g., has a longer wakeup latency, lessfully enabled). Examples of power savings actions that a core 102 mayundergo are stopping execution of instructions, stopping clocks to,lowering voltages to, and/or removing power from portions of the core(e.g., functional units and/or local cache) or to the entire core.

Additionally, the microprocessor 100 is configured to perform powerreduction actions that are trans-core. The trans-core power reductionsactions implicate, or affect, more than one core 102 of themicroprocessor 100. For example, the shared cache 119 may be large andconsume a relatively large amount of power; thus, significant powersavings may be achieved by removing the clock signal and/or power to theshared cache 119. However, in order to remove the clock or power to theshared cache 119, all of the cores 102 sharing the cache must agree sothat data coherency is maintained. Embodiments are contemplated in whichthe microprocessor 100 includes other shared power-related resources,such as shared clock and power sources. In one embodiment, themicroprocessor 100 is coupled to a chipset of the system that includes amemory controller, peripheral controllers and/or power managementcontroller. In other embodiments, one or more of the controllers areintegrated within the microprocessor 100. System power savings may beachieved by the microprocessor 100 informing the controllers that ittook an action that enables the controllers to take power savingactions. For example, the microprocessor 100 may inform the controllersthat it invalidated and turned off the caches of the microprocessor suchthat they need not be snooped.

In addition to the notion of a core C-state, there is the notion of a“package” C-state (also referred to as a packet idle state or packagesleep state) for the microprocessor 100 as a whole. The package C-statecorresponds to the lowest (i.e., highest-power-consuming) common coreC-state of the cores 102 (see, for example, field 246 of FIG. 2 andblock 318 of FIG. 3). However, the package C-state involves themicroprocessor 100 performing one or more trans-core power reductionactions in addition to the core-specific power reduction actions. Anexample of trans-core power savings actions that may be associated withpackage C-states include turning off a phase-locked-loop (PLL) thatgenerates clock signals and flushing the shared cache 119 and stoppingits clocks and/or power, which enables the memory/peripheral controllerto refrain from snooping the local and shared microprocessor 100 caches.Other examples are changing voltage, frequency and/or bus clock ratio;reducing the size of cache memories, such as the shared cache 119; andrunning the shared cache 119 at half speed.

In many cases, the operating system is effectively relegated toexecuting instructions on individual cores 102 and can therefore putindividual cores to sleep (e.g., into core C-states), but does not havea means to directly put the microprocessor 100 package to sleep (e.g.,into package C-states). Advantageously, embodiments are described inwhich the cores 102 of the microprocessor 100 work cooperatively, withthe help of the control unit 104, to detect when all cores 102 haveentered a core C-state and are ready to allow trans-core power savingsactions to occur.

Referring now to FIG. 9, a flowchart illustrating operation of themicroprocessor 100 to enter a low power package C-state is shown. Theembodiment of FIG. 9 is described using the example of the execution ofMWAIT instructions in which the microprocessor 100 is coupled to achipset. However, it should be understood that in other embodiments theoperating system employs other power management instructions and themaster core 102 communicates with controllers that are integrated withinthe microprocessor 100 and that employ a different handshake protocolthan described. The operation is described from the perspective of asingle core, but each of the cores 102 of the microprocessor 100 mayencounter an MWAIT instruction and operate according to the descriptionto collectively cause the microprocessor 100 to enter the optimizedstate. Flow begins at block 902.

At block 902, a core 102 encounters an MWAIT instruction that specifiesa target C-state, denoted Cx in FIG. 9, where x is a non-negativeinteger value. Flow proceeds to block 904.

At block 904, the core 102 writes a sync request to its sync register108 with the C bit 224 set and a C-state field 226 value of x (denotedsync Cx in FIG. 9). Additionally, the sync request specifies in itswakeup events field 204 that the core 102 is to be awakened on allwakeup events. As a result, the control unit 104 puts the core 102 tosleep. Preferably, the core 102 writes back and invalidates its localcaches before it writes the sync Cx. Flow proceeds to block 906.

At block 906, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync Cx. As described above, the x valuewritten by the other cores 102 may be different, and the control unit104 posts the lowest common C-state value to the lowest common C-statefield 246 of the status word 242 of the status register 106 (per block318). Prior to block 906, while the core 102 is asleep, it may beawakened by a wakeup event, such as an interrupt (e.g., at blocks 305and 306). More specifically, there is no guarantee that the operatingsystem will execute an MWAIT for all of the cores 102, which would allowthe microprocessor 100 to perform power savings actions associated witha package C-state, before a wakeup event occurs (e.g., interrupt)directed to one of the cores 102 that effectively cancels the MWAITinstruction. However, once the core 102 is awakened at block 906, thecore 102 is (indeed, all the cores 102 are) still executing microcode asa result of the MWAIT instruction (at block 902) and remains inmicrocode, during which time interrupts are disabled (i.e., themicrocode does not allow itself to be interrupted), until block 924. Inother words, while less than all of the cores 102 have received an MWAITinstruction to go to sleep, individual cores 102 may sleep, but themicroprocessor 100 as a package does not indicate to the chipset that itis ready to enter a package sleep state; however, once all the cores 102have agreed to enter a package sleep state, which is effectivelyindicated by the sync condition occurrence at block 906, the master core102 is allowed to complete a package sleep state handshake protocol withthe chipset (e.g., blocks 908, 909 and 921 below) without beinginterrupted and without any of the other cores 102 being interrupted.Flow proceeds to decision block 907.

At decision block 907, the core 102 determines whether it is the mastercore 102 of the microprocessor 100. Preferably, a core 102 is the mastercore 102 if it determines it is the BSP at reset time. If the core 102is the master, flow proceeds to block 908; otherwise, flow proceeds toblock 914.

At block 908, the master core 102 writes back and invalidates the sharedcache 119 and then communicates to the chipset that it may takeappropriate actions that may reduce power consumption. For example, thememory controller and/or peripheral controller may refrain from snoopingthe local and shared caches of the microprocessor 100 since they allremain invalid while the microprocessor 100 is in the package C-state.For another example, the chipset may signal to the microprocessor 100 tocause the microprocessor 100 to take power savings actions (e.g., assertx86-style STPCLK, SLP, DPSLP, NAP, VRDSLP signals as described below).Preferably, the core 102 communicates power management information basedon the lowest common C-state field 246 value. In one embodiment, thecore 102 issues an I/O Read bus cycle to an I/O address that providesthe chipset the relevant power management information, e.g., the packageC-state state value. Flow proceeds to block 909.

At block 909, the master core 102 waits for the chipset to assert theSTPCLK signal. Preferably, if the STPCLK signal is not asserted after apredetermined number of clock cycles, the control unit 104 detects thiscondition and wakes up all the cores 102 after killing their pendingsync requests and indicates the error in the error code field 248. Flowproceeds to block 914.

At block 914, the core 102 writes a sync 14. In one embodiment, the syncrequest specifies in its wakeup events field 204 that the core 102 is tonot be awakened on any wakeup event. As a result, the control unit 104puts the core 102 to sleep. Flow proceeds to block 916.

At block 916, the core 102 gets awakened by the control unit 104 whenall cores 102 have written a sync 14. Flow proceeds to decision block919.

At decision block 919, the core 102 determines whether it is the mastercore 102 of the microprocessor 100. If so, flow proceeds to block 921;otherwise, flow proceeds to block 924.

At block 921, the master core 102 issues a stop grant cycle to thechipset on the microprocessor 100 bus to notify the chipset that it maytake trans-core, i.e., package-wide, power savings actions regarding themicroprocessor 100 package as a whole, e.g., refrain from snooping thecaches of the microprocessor 100, remove the bus clock (e.g., x86-styleBCLK) to the microprocessor 100, and assert other signals (e.g.,x86-style SLP, DPSLP, NAP, VRDSLP) on the bus to cause themicroprocessor 100 to remove clocks and/or power to various portions ofthe microprocessor 100. Although embodiments are described herein thatinvolve a handshake protocol between the microprocessor 100 and achipset involving the I/O read (at block 908), the assertion of STPCLK(at block 909) and the issuing of the stop grant cycle (at block 921)which are historically associated with x86 architecture-based systems,it should be understood that other embodiments are contemplated thatinvolve systems with other instruction set architecture-based systemswith different protocols but in which it is also desirable to savepower, increase performance and/or reduce complexity. Flow proceeds toblock 924.

At block 924, the core 102 writes a sleep request to the sync register108, i.e., with the sleep bit 212 set and the S bit 222 clear.Additionally, the sync request specifies in its wakeup events field 204that the core 102 is to be awakened only on the wakeup event of thede-assertion of STPCLK. As a result, the control unit 104 puts the core102 to sleep. Flow ends at block 924.

Referring now to FIG. 10, a timing diagram illustrating an example ofthe operation of the microprocessor 100 according to the flowchart ofFIG. 9 is shown. In the example, a configuration of a microprocessor 100with three cores 102, denoted core 0, core 1 and core 2, is shown;however, it should be understood that in other embodiments themicroprocessor 100 may include different numbers of cores 102.

Core 0 encounters an MWAIT instruction specifying C-state 4 (per block902). Core 0 then writes a sync C4 and is put to sleep (per block 904).Core 1 encounters an MWAIT instruction specifying C-state 3 (per block902). Core 1 then writes a sync C3 and is put to sleep (per block 904).Core 2 encounters an MWAIT instruction specifying C-state 2 (per block902). Core 2 then writes a sync C2 and is put to sleep (per block 904).As shown, the time at which each of the cores writes the sync Cx mayvary. Indeed, it is possible that one or more of the cores may notencounter an MWAIT instruction before some other event occurs, such asan interrupt.

When all the cores have written the sync Cx, the control unit 104 wakesthem all up simultaneously (per block 906). The master core then issuesthe I/O Read bus cycle (per block 908) and waits for the assertion ofSTPCLK (per block 909). All of the cores write a sync 14 and are put tosleep (per block 914). Since only the master core flushes the sharedcache 119, issues the I/O Read bus cycle and waits for the assertion ofSTPCLK, the time at which each of the cores writes the sync 14 may vary,as shown. Indeed, the master core may write the sync 14 on the order ofhundreds of microseconds after the other cores.

When all the cores have written the sync 14, the control unit 104 wakesthem all up simultaneously (per block 916). Only the master core issuesthe stop grant cycle (per block 921). All of the cores write a sleeprequest waiting on the de-assertion of STPCLK and are put to sleep (perblock 924). Since only the master core issues the stop grant cycle, thetime at which each of the cores writes the sleep request may vary, asshown.

When STPCLK is de-asserted, the control unit 104 wakes up all the cores.

As may be observed from FIG. 10, advantageously core 1 and core 2 areable to sleep for a significant portion of the time while core 0performs the handshake protocol. However, it is noted that the amount oftime required to wake up the microprocessor 100 from the package sleepstate is generally proportional to how deep the sleep is (i.e., howgreat the power savings while in the sleep state). Consequently, incases where the package sleep state is relatively deep (or even where anindividual core 102 sleep state is relatively deep), it may be desirableto even further reduce the wakeup occurrences and/or time required towakeup associated with the handshake protocol. FIG. 11 describes anembodiment in which a single core 102 handles the handshake protocolwhile the other cores 102 continue to sleep. Furthermore, according tothe embodiment of FIG. 11, further power savings may be obtained byreducing the number of cores 102 that are awakened in response to awakeup event.

Referring now to FIG. 11, a flowchart illustrating operation of themicroprocessor 100 to enter a low power package C-state according to analternate embodiment is shown. The embodiment of FIG. 11 is describedusing the example of the execution of MWAIT instructions in which themicroprocessor 100 is coupled to a chipset. However, it should beunderstood that in other embodiments the operating system employs otherpower management instructions and the last-syncing core 102 communicateswith controllers that are integrated within the microprocessor 100 andthat employ a different handshake protocol than described. Theembodiment of FIG. 11 is similar in some respects to the embodiment ofFIG. 9. However, the embodiment of FIG. 11 is designed to facilitatepotentially greater power savings in the presence of an environment inwhich the operating system requests the microprocessor 100 to enter verylow power states and tolerates the latencies associated with them. Morespecifically, the embodiment of FIG. 11 facilitates gating power to thecores and waking up only one of the cores when necessary, such as tohandle interrupts, for example. Embodiments are contemplated in whichthe microprocessor 100 supports operation in both the mode of FIG. 9 andthe mode of FIG. 11. Furthermore, the mode may be configurable, eitherin manufacturing (e.g., by fuses 114) and/or via software control orautomatically decided by the microprocessor 100 depending on theparticular C-state specified by the MWAIT instructions. Flow begins atblock 1102.

At block 1102, a core 102 encounters an MWAIT instruction that specifiesa target C-state, denoted Cx in FIG. 11. Flow proceeds to block 1104.

At block 1104, the core 102 writes a sync request to its sync register108 with the C bit 224 set and a C-state field 226 value of x (denotedsync Cx in FIG. 11). The sync request also sets the sel wake bit 214 andthe PG bit 208. Additionally, the sync request specifies in its wakeupevents field 204 that the core 102 is to be awakened on all wakeupevents except assertion of STPCLK and deassertion of STPCLK (˜STPCLK).(Preferably, there are other wakeup events, such as AP startup, forwhich the sync request specifies the core 102 is not to be awakened.) Asa result, the control unit 104 puts the core 102 to sleep, whichincludes refraining from providing power to the core 102 because the PGbit 208 was set. Additionally, the core 102 writes back and invalidatesits local cache memories and saves (preferably to the PRAM 116) its core102 state before writing the sync request. The core 102 will restore itsstate (e.g., from the PRAM 116) when it is subsequently awakened (e.g.,at block 1137, 1132 or 1106). As described above, particularly withrespect to FIG. 3, when the last core 102 writes its sync request withthe sel wake bit 214 set, the control unit 104 automatically blocks offall wakeup events for all cores 102 other than the last writing core 102(per block 326). Flow proceeds to block 1106.

At block 1106, the control unit 104 awakens the last writing core 102when all cores 102 have written a sync Cx. As described above, thecontrol unit 104 keeps the S bit 222 set for the other cores 102 eventhough it wakes up the last writing core 102 and clears its S bit 222.Prior to block 1106, while the core 102 is asleep, it may be awakened bya wakeup event, such as an interrupt. However, once the core 102 isawakened at block 1106, the core 102 is still executing microcode as aresult of the MWAIT instruction (at block 1102) and remains inmicrocode, during which time interrupts are disabled (i.e., themicrocode does not allow itself to be interrupted), until block 1124. Inother words, while less than all of the cores 102 have received an MWAITinstruction to go to sleep, only individual cores 102 will sleep, butthe microprocessor 100 as a package does not indicate to the chipsetthat it is ready to enter a package sleep state; however, once all thecores 102 have agreed to enter a package sleep state, which is indicatedby the sync condition occurrence at block 1106, the core 102 awakened atblock 906 (the last writing core 102, which caused the sync conditionoccurrence) is allowed to complete the package sleep state handshakeprotocol with the chipset (e.g., blocks 1108, 1109 and 1121 below)without being interrupted and without any of the other cores 102 beinginterrupted. Flow proceeds to block 1108.

At block 1108, the core 102 writes back and invalidates the shared cache119 and then communicates to the chipset that it may take appropriateactions that may reduce power consumption. Flow proceeds to block 1109.

At block 1109, the core 102 waits for the chipset to assert the STPCLKsignal. Preferably, if the STPCLK signal is not asserted after apredetermined number of clock cycles, the control unit 104 detects thiscondition and wakes up all the cores 102 after killing their pendingsync requests and indicates the error in the error code field 248. Flowproceeds to block 1121.

At block 1121, the core 102 issues a stop grant cycle to the chipset onthe bus. Flow proceeds to block 1124.

At block 1124, the core 102 writes a sleep request to the sync register108, i.e., with the sleep bit 212 set and the S bit 222 clear, and withthe PG bit 208 set. Additionally, the sync request specifies in itswakeup events field 204 that the core 102 is to be awakened only on thewakeup event of the de-assertion of STPCLK. As a result, the controlunit 104 puts the core 102 to sleep. Flow proceeds to block 1132.

At block 1132, the control unit 104 detects the de-assertion of STPCLKand wakes up the core 102. It is noted that prior to the control unit104 waking up the core 102, the control unit 104 also un-gates power tothe core 102. Advantageously, at this point the core 102 is the onlyrunning core 102, which provides an opportunity for the core 102 toperform any actions that must be performed while no other cores 102 arerunning Flow proceeds to block 1134.

At block 1134, the core 102 writes to a register (not shown) in thecontrol unit 104 to unblock the wakeup events for each of the othercores 102 specified in the wakeup events field 204 of their respectivesync register 108. Flow proceeds to block 1136.

At block 1136, the core 102 handles any pending wakeup events directedto it. For example, in one embodiment the system comprising themicroprocessor 100 permits both directed interrupts (i.e., interruptsdirected to a specific core 102 of the microprocessor 100) andnon-directed interrupts (i.e., interrupts that may be handled by anycore 102 of the microprocessor 100 as the microprocessor 100 selects).An example of a non-directed interrupt is what is commonly referred toas a “low priority interrupt.” In one embodiment, the microprocessor 100advantageously directs non-directed interrupts to the single core 102that is awakened at the de-assertion of STPCLK at block 1132 since it isalready awake and can handle the interrupt in hopes that the other cores102 do not have any pending wakeup events and can therefore continue tosleep and be power-gated. Flow returns to block 1104.

If no specified wakeup events are pending for a core 102 other than thecore 102 that was awakened at block 1132 when the wakeup events areunblocked at block 1134, then advantageously the core 102 will continueto sleep and be power-gated per block 1104. However, if a specifiedwakeup event is pending for the core 102 when wakeup events areunblocked at block 1134, then the core 102 will be un-power-gated andawakened by the control unit 104. In this case, a different flow beginsat block 1137 of FIG. 11.

At block 1137, another core 102 (i.e., a core 102 other than the core102 that unblocks the wakeup events at block 1134) is awakened after thewakeup events are unblocked at block 1134. The other core 102 handlesany pending wakeup events directed to it, e.g., handles an interrupt.Flow proceeds from block 1137 to block 1104.

Referring now to FIG. 12, a timing diagram illustrating an example ofthe operation of the microprocessor 100 according to the flowchart ofFIG. 11 is shown. In the example, a configuration of a microprocessor100 with three cores 102, denoted core 0, core 1 and core 2, is shown;however, it should be understood that in other embodiments themicroprocessor 100 may include different numbers of cores 102.

Core 0 encounters an MWAIT instruction specifying C-state 7 (per block1102). In the example, C-state 7 permits power-gating. Core 0 thenwrites a sync C7 with the sel wake bit 214 set (indicated by “SW” inFIG. 12) and the PG bit 208 set, and is put to sleep and power-gated(per block 1104). Core 1 encounters an MWAIT instruction specifyingC-state 7 (per block 1102). Core 1 then writes a sync C7 with the selwake bit 214 set and the PG bit 208 set, and is put to sleep andpower-gated (per block 1104). Core 2 encounters an MWAIT instructionspecifying C-state 7 (per block 1102). Core 2 then writes a sync C7 withthe sel wake bit 214 set and the PG bit 208 set, and is put to sleep andpower-gated (per block 1104) (however, in an optimization embodimentdescribed at block 314, the last-writing core is not power-gated). Asshown, the time at which each of the cores writes the sync C7 may vary.

When the last core writes the sync C7 with the sel wake bit 214 set, thecontrol unit 104 blocks off the wakeup events for all but thelast-writing core (per block 326), which in the example of FIG. 12 iscore 2. Additionally, the control unit 104 wakes up only thelast-writing core (per block 1106), which may result in power savingsbecause the other cores continue to sleep and be power-gated while core2 performs the handshake protocol with the chipset. Core 2 then issuesthe I/O Read bus cycle (per block 1108) and waits for the assertion ofSTPCLK (per block 1109). In response to STPCLK, core 2 issues the stopgrant cycle (per block 1121) and writes a sleep request with the PG bit208 set waiting on the de-assertion of STPCLK and is put to sleep andpower-gated (per block 1124). The cores may sleep and be power-gated fora relatively long time.

When STPCLK is de-asserted, the control unit 104 wakes up only core 2(per block 1132). In the example of FIG. 12, the chipset de-assertsSTPCLK in response to reception of a non-directed interrupt, which itforwards to the microprocessor 100. The microprocessor 100 directs thenon-directed interrupt to core 2, which may result in power savingsbecause the other cores continue to sleep and be power-gated. Core 2unblocks the wakeup events of the other cores (per block 1134) andservices the non-directed interrupt (per block 1136). Core 2 then againwrites a sync C7 with the sel wake bit 214 set and the PG bit 208 set,and is put to sleep and power-gated (per block 1104).

When core 2 writes the sync C7 with the sel wake bit 214 set, thecontrol unit 104 blocks off the wakeup events for all but core 2, i.e.,the last-writing core (per block 326) since the sync requests for theother cores are still pending, i.e., the S bits 222 of the other coreswere not cleared by the wakeups of core 2. Additionally, the controlunit 104 wakes up only core 2 (per block 1106). Core 2 then issues theI/O Read bus cycle (per block 1108) and waits for the assertion ofSTPCLK (per block 1109). In response to STPCLK, core 2 issues the stopgrant cycle (per block 1121) and writes a sleep request with the PG bit208 set waiting on the de-assertion of STPCLK and is put to sleep andpower-gated (per block 1124).

When STPCLK is de-asserted, the control unit 104 wakes up only core 2(per block 1132). In the example of FIG. 12, STPCLK is de-assertedbecause of another non-directed interrupt; therefore, the microprocessor100 directs the interrupt to core 2, which may result in power savings.Core 2 again unblocks the wakeup events of the other cores (per block1134) and services the non-directed interrupt (per block 1136). Core 2then again writes a sync C7 with the sel wake bit 214 set and the PG bit208 set, and is put to sleep and power-gated (per block 1104).

This cycle may continue for a relatively lengthy time, namely, as longas only non-directed interrupts are generated. FIG. 13 depicts anexample of the handling of interrupts directed to a different core otherthan the last-writing core.

As may be observed by comparing FIG. 10 and FIG. 12, advantageously inthe embodiment of FIG. 12, once the cores 102 initially go to sleep(after writing the sync C7 in the example of FIG. 12), only one of thecores 102 is awakened again to perform the handshaking protocol with thechipset and the other cores 102 remain asleep, which may be asignificant advantage if the cores 102 were in a relatively deep sleepstate. The power savings may be significant, particularly in cases wherethe operating system recognizes the workload on the system issufficiently small for a single core 102 to handle the workload.

Furthermore, advantageously, only one of the cores 102 is awakened (toservice non-directed events such as a low priority interrupt), as longas no wakeup events are directed to the other cores 102. Again, this maybe a significant advantage if the cores 102 were in a relatively deepsleep state. The power savings may be significant, particularly insituations where there is effectively no workload on the system exceptrelatively infrequent non-directed interrupts, such as USB interrupts.Still further, even if a wakeup event occurs that is directed to anothercore 102 (e.g., interrupts that the operating system directs to a singlecore 102, such as operating system timer interrupts), advantageously theembodiments may dynamically switch the single core 102 that performs thepackage sleep state protocol and services non-directed wakeup events, asillustrated in FIG. 13, so that the benefit of waking up only a singlecore 102 are enjoyed.

Referring now to FIG. 13, a timing diagram illustrating an alternateexample of the operation of the microprocessor 100 according to theflowchart of FIG. 11 is shown. The example of FIG. 13 is similar in manyrespects to the example of FIG. 12; however, at the point where STPCLKis de-asserted in the first instance, the interrupt is a directedinterrupt to core 1 (rather than a non-directed interrupt as in theexample of FIG. 12). Consequently, the control unit 104 wakes up core 2(per block 1132), and subsequently wakes up core 1 after the wakeupevents are unblocked (per block 1134) by core 2. Core 2 then againwrites a sync C7 with the sel wake bit 214 set and the PG bit 208 set,and is put to sleep and power-gated (per block 1104).

Core 1 services the directed interrupt (per block 1137). Core 1 thenagain writes a sync C7 with the sel wake bit 214 set and the PG bit 208set, and is put to sleep and power-gated (per block 1104). In theexample, core 2 wrote its sync C7 before core 1 wrote its sync C7.Consequently, although core 0 still has its S bit 222 set when it wroteits initial sync C7, core 1's S bit 222 was cleared when it wasawakened. Therefore, when core 2 wrote the sync C7 after unblocking thewakeup events, it was not the last core to write the sync C7 request;rather, core 1 became the last core to write the sync C7 request.

When core 1 writes the sync C7 with the sel wake bit 214 set, thecontrol unit 104 blocks off the wakeup events for all but core 1, i.e.,the last-writing core (per block 326) since the sync requests for core 0is still pending, i.e., it was not cleared by the wakeups of core 1 andcore 2, and core 2 has already (in the example) written the sync 14request. Additionally, the control unit 104 wakes up only core 1 (perblock 1106). Core 1 then issues the I/O Read bus cycle (per block 1108)and waits for the assertion of STPCLK (per block 1109). In response toSTPCLK, core 1 issues the stop grant cycle (per block 1121) and writes asleep request with the PG bit 208 set waiting on the de-assertion ofSTPCLK and is put to sleep and power-gated (per block 1124).

When STPCLK is de-asserted, the control unit 104 wakes up only core 1(per block 1132). In the example of FIG. 12, STPCLK is de-assertedbecause of a non-directed interrupt; therefore, the microprocessor 100directs the interrupt to core 1, which may result in power savings. Thecycle of handling non-directed interrupts by core 1 may continue for arelatively lengthy time, namely, as long as only non-directed interruptsare generated. In this manner, the microprocessor 100 advantageously maysave power by directing non-directed interrupts to the core 102 to whichthe most recent interrupt was directed, which in the example of FIG. 13involved switching to a different core. Core 1 again unblocks the wakeupevents of the other cores (per block 1134) and services the non-directedinterrupt (per block 1136). Core 1 then again writes a sync C7 with thesel wake bit 214 set and the PG bit 208 set, and is put to sleep andpower-gated (per block 1104).

It should be understood that although embodiments have been described inwhich the power management instruction is an x86 MWAIT instruction,other embodiments are contemplated in which sync requests are employedto perform other power management instructions. For example, themicroprocessor 100 may perform similar actions in response to a readfrom a set of predetermined I/O port addresses associated with thevarious C-states. For another example, the power management instructionmay be from a different instruction set architecture than the x86architecture.

Dynamic Reconfiguration of Multi-Core Processor

Each core 102 of the microprocessor 100 generates configuration-relatedvalues based on the configuration of cores 102 of the microprocessor100. Preferably, microcode of each core 102 generates, saves and usesthe configuration-related values. Embodiments are described in which thegeneration of the configuration-related values advantageously may bedynamic as described below. Examples of configuration-related valuesinclude, but are not limited to, the following.

Each core 102 generates a global core number described above withrespect to FIG. 2. The global core number indicates the core number ofthe core 102 globally relative to all the cores 102 of themicroprocessor 100 in contrast to the local core number 256 thatindicates the core number of the core 102 locally relative to only thecores 102 of the die 406 on which the core 102 resides. In oneembodiment, the core 102 generates the global core number as the sum ofthe product of its die number 258 and the number of cores 102 per dieand its local core number 256, as shown here:

global core number=(die number*number cores per die)+local core number.

Each core 102 also generates a virtual core number. The virtual corenumber is the global core number minus the number of disabled cores 102having a global core number lower than the global core number of theinstant core 102. Thus, in the case in which all cores 102 of themicroprocessor 100 are enabled, the global core number and the virtualcore number are the same. However, if one or more of the cores 102 aredisabled, leaving holes, the virtual core number of a core 102 may bedifferent from its global core number. In one embodiment, each core 102populates the APIC ID field in its corresponding APIC ID register withits virtual core number. However, according to alternate embodiments(e.g., FIGS. 22 and 23) this is not the case. Furthermore, in oneembodiment, the operating system may update the APIC ID in the APIC IDregister.

Each core 102 also generates a BSP flag, which indicates whether thecore 102 is the BSP. In one embodiment, normally (e.g., when the “allcores BSP” feature of FIG. 23 is disabled) one core 102 designatesitself the bootstrap processor (BSP) and each of the other cores 102designates itself as an application processor (AP). After a reset, theAP cores 102 initialize themselves and then go to sleep waiting for theBSP to tell them to begin fetching and executing instructions. Incontrast, after initializing itself, the BSP core 102 immediately beginsfetching and executing instructions of the system firmware, e.g., BIOSbootstrap code, which initializes the system (e.g., verifies that thesystem memory and peripherals are working properly and initializesand/or configures them) and bootstraps the operating system, i.e., loadsthe operating system (e.g., from disk) and transfers control to theoperating system. Prior to bootstrapping the operating system, the BSPdetermines the system configuration (e.g., the number of cores 102 orlogical processors in the system) and saves it in memory so that theoperating system may read it after it is booted. After beingbootstrapped, the operating system instructs the AP cores 102 to beginfetching and executing instructions of the operating system. In oneembodiment, normally (e.g., when the “modify BSP” and “all cores BSP”features of FIGS. 22 and 23, respectively, are disabled) a core 102designates itself the BSP if its virtual core number is zero, and allother cores 102 designate themselves an AP core 102. Preferably, a core102 populates the BSP flag bit in the APIC base address register of itscorresponding APIC with its BSP flag configuration-related value.According to one embodiment, as described above, the BSP is master core102 of blocks 907 and 919 that performs the package sleep statehandshake protocol of FIG. 9.

Each core 102 also generates an APIC base value for populating the APICbase register. The APIC base address is generated based on the APIC IDof the core 102. In one embodiment, the operating system may update theAPIC base address in the APIC base address register.

Each core 102 also generates a die master indicator, which indicateswhether the core 102 is the master core 102 of the die 406 that includesthe core 102.

Each core 102 also generates a slice master indicator, which indicateswhether the core 102 is the master core 102 of the slice that includesthe instant core 102, assuming the microprocessor 100 is configured withslices, which are described above.

Each core 102 computes the configuration-related values and operatesusing the configuration-related values so that the system comprising themicroprocessor 100 operates correctly. For example, the system directsinterrupt requests to the cores 102 based on their associated APIC IDs.The APIC ID determines which interrupt requests the core 102 willrespond to. More specifically, each interrupt request includes adestination identifier, and a core 102 responds to an interrupt requestonly if the destination identifier matches the APIC ID of the core 102(or if the interrupt request identifier is a special value thatindicates it is a request for all cores 102). For another example, eachcore 102 must know whether it is the BSP so that it executes the initialBIOS code and bootstraps the operating system and in one embodimentperforms the package sleep state handshake protocol as described withrespect to FIG. 9. Embodiments are described below (see FIGS. 22 and 23)in which the BSP flag and APIC ID may be altered from their normalvalues for specific purposes, such as for testing and/or debug.

Referring now to FIG. 14, a flowchart illustrating dynamicreconfiguration of the microprocessor 100 is shown. In the descriptionof FIG. 14 reference is made to the multi-die microprocessor 100 of FIG.4, which includes two die 406 and eight cores 102. However, it should beunderstood that the dynamic reconfiguration described may apply to amicroprocessor 100 with a different configuration, namely with more thantwo dies or a single die, and more or less than eight cores 102 but atleast two cores 102. The operation is described from the perspective ofa single core, but each of the cores 102 of the microprocessor 100operates according to the description to collectively dynamicallyreconfigure the microprocessor 100. Flow begins at block 1402.

At block 1402, the microprocessor 100 is reset and hardware of themicroprocessor 100 populates the configuration register 112 of each core102 with the appropriate values based on the number of enabled cores 102and the die number on which the control unit 104 resides. In oneembodiment, the local core number 256 and die number 258 are hardwired.As described above, the hardware may determine whether a core 102 isenabled or disabled from the blown or unblown state of fuses 114. Flowproceeds to block 1404.

At block 1404, the core 102 reads the configuration word 252 from theconfiguration register 112. The core 102 then generates itsconfiguration-related values based on the value of the configurationword 252 read at block 1402. In the case of a multi-die microprocessor100 configuration, the configuration-related values generated at block1404 will not take into account the cores 102 of the other die 406;however, the configuration-related values generated at blocks 1414 and1424 (as well as block 1524 of FIG. 15) will take into account the cores102 of the other die 406, as described below. Flow proceeds to block1406.

At block 1406, the core 102 causes the enabled bit 254 values of thelocal cores 102 in the local configuration register 112 to be propagatedto the corresponding enabled bits 254 of configuration register 112 ofthe remote die 406. For example, with respect to the configuration ofFIG. 4, a core 102 on die A 406A causes the enabled bits 254 associatedwith cores A, B, C and D (local cores) in the configuration register 112of die A 406A (local die) to be propagated to the enabled bits 254associated with cores A, B, C and D in the configuration register 112 ofdie B 406B (remote die); conversely, a core 102 on die B 406B causes theenabled bits 254 associated with cores E, F, G and H (local cores) inthe configuration register 112 of die B 406B (local die) to bepropagated to the enabled bits 254 associated with cores E, F, G and Hin the configuration register 112 of die A 406A (remote die). In oneembodiment, the core 102 causes the propagation to the other die 406 bywriting to the local configuration register 112. Preferably, the writeby the core 102 to the local configuration register 112 causes no changeto the local configuration register 112, but causes the local controlunit 104 to propagate the local enabled bit 254 values to the remote die406. Flow proceeds to block 1408.

At block 1408, the core 102 writes a sync request to its sync register108 with a sync condition value of 8 (denoted sync 8 in FIG. 14). As aresult, the control unit 104 puts the core 102 to sleep. Flow proceedsto block 1412.

At block 1412, the control unit 104 awakens the core 102 when allenabled cores 102 in the set of cores specified by the core set field228 have written a sync 8. It is noted that in the case of a multi-die406 microprocessor 100 configuration, the sync condition occurrence maybe a multi-die sync condition occurrence. That is, the control unit 104will wait to wakeup (or interrupt in the case where the cores 102 havenot set the sleep bit 212 and thereby have elected not to sleep) thecores 102 until all the cores 102 specified in the core set field 228(which may include cores 102 on both dies 406) and which are enabled (asindicated by the enabled bits 254) have written their sync request. Flowproceeds to block 1414.

At block 1414, the core 102 again reads the configuration register 112and generates its configuration-related values based on the new value ofthe configuration word 252 that includes the correct values of theenabled bits 254 from the remote die 406. Flow proceeds to decisionblock 1416.

At decision block 1416, the core 102 determines whether it shoulddisable itself. In one embodiment, the core 102 decides that it needs todisable itself because fuses 114 have been blown that the microcodereads (prior to decision block 1416) in its reset processing thatindicate that the core 102 should disable itself The fuses 114 may beblown during or after manufacturing of the microprocessor 100.Alternatively, updated fuse 114 values may be scanned into holdingregisters, as described above, and the scanned in values indicate to thecore 102 that it should disable itself. FIG. 15 describes an alternateembodiment in which a core 102 determines by a different manner that itshould disable itself. If the core 102 determines that it should disableitself, flow proceeds to block 1417; otherwise, flow proceeds to block1418.

At block 1417, the core 102 writes the disable core bit 236 to causeitself to be removed from the list of enabled cores 102, e.g., to haveits corresponding enabled bit 254 in the configuration word 252 of theconfiguration register 112 cleared. Afterward, the core 102 preventsitself from executing any more instructions, preferably by setting oneor more bits to turn off its clock signals and have its power removed.Flow ends at block 1417.

At block 1418, the core 102 writes a sync request to its sync register108 with a sync condition value of 9 (denoted sync 9 in FIG. 14). As aresult, the control unit 104 puts the core 102 to sleep. Flow proceedsto block 1422.

At block 1422, the core 102 gets awakened by the control unit 104 whenall enabled cores 102 have written a sync 9. Again, in the case of amulti-die 406 microprocessor 100 configuration, the sync conditionoccurrence may be a multi-die sync condition occurrence based on theupdated values in the configuration register 112. Furthermore, thecontrol unit 104 will exclude from consideration the core 102 thatdisabled itself at block 1417 as it determines whether a sync conditionoccurred. More specifically, in the circumstances in which all the othercores 102 (other than the core 102 disabling itself) write a sync 9before the disabling-itself core 102 writes the sync register 108 atblock 1417, then the control unit 104 will detect the sync conditionoccurred (at block 316) when the disabling-itself core 102 writes thesync register 108 with the disable core bit 236 set at block 1417. Thisis because at that point the control unit 104 no longer considers thedisabled core 102 when determining whether the sync condition hasoccurred because the enabled bit 254 of the disabled core 102 is clear.That is, the control unit 104 determines that the sync condition hasoccurred because all of the enabled cores 102, which does not includethe disabled core 102, have written a sync 9, regardless of whether thedisabled core 102 has written a sync 9. Flow proceeds to block 1424.

At block 1424, the core 102 again reads the configuration register 112,and the new value of the configuration word 252 reflects a disabled core102 if one was disabled by the operation of block 1417 by another core102. The core 102 then again generates its configuration-related values,similar to the manner at block 1414, based on the new value of theconfiguration word 252. The presence of a disabled core 102 may causesome of the configuration-related values to be different than the valuesgenerated at block 1414. For example, as described above, the virtualcore number, APIC ID, BSP flag, BSP base address, die master and slicemaster may change due to the presence of a disabled core 102. In oneembodiment, after generating the configuration-related values, one ofthe cores 102 (e.g., BSP) writes to the uncore PRAM 116 some of theconfiguration-related values that are global to all cores 102 of themicroprocessor 100 so that they may be subsequently read by all thecores 102. For example, in one embodiment the global theconfiguration-related values are read by a core 102 to perform anarchitectural instruction (e.g., the x86 CPUID instruction) thatrequests global information about the microprocessor 100, such as thenumber of cores 102 of the microprocessor 100. Flow proceeds to decisionblock 1426.

At block 1426, the core 102 comes out of reset and begins fetchingarchitectural instructions. Flow ends at block 1426.

Referring now to FIG. 15, a flowchart illustrating dynamicreconfiguration of the microprocessor 100 according to an alternateembodiment is shown. In the description of FIG. 15 reference is made tothe multi-die microprocessor 100 of FIG. 4, which includes two die 406and eight cores 102. However, it should be understood that the dynamicreconfiguration described may apply to a microprocessor 100 with adifferent configuration, namely with more than two dies or a single die,and more or less than eight cores 102 but at least two cores 102. Theoperation is described from the perspective of a single core, but eachof the cores 102 of the microprocessor 100 operates according to thedescription to collectively dynamically reconfigure the microprocessor100. More specifically, FIG. 15 describes the operation of one core 102that encounters the core disable instruction, whose flow begins at block1502, and the operation of the other cores 102, whose flow begins atblock 1532.

At block 1502, one of the cores 102 encounters an instruction thatinstructs the core 102 to disable itself. In one embodiment, theinstruction is an x86 WRMSR instruction. In response, the core 102 sendsa reconfigure message to the other cores 102 and sends them aninter-core interrupt. Preferably, the core 102 traps to microcode inresponse to the instruction to disable itself (at block 1502) or inresponse to the interrupt (at block 1532) and remains in microcode,during which time interrupts are disabled (i.e., the microcode does notallow itself to be interrupted), until block 1526. Flow proceeds fromblock 1502 to block 1504.

At block 1532, one of the other cores 102 (i.e., a core 102 other thanthe core 102 that encountered the disable instruction at block 1502)gets interrupted and receives the reconfigure message as a result of theinter-core interrupt sent at block 1502. As described above, althoughflow at block 1532 is described from the perspective of a single core102, each of the other cores 102 (i.e., not the core 102 at block 1502)gets interrupted and receives the message at block 1532 and performs thesteps at blocks 1504 through 1526. Flow proceeds from block 1532 toblock 1504.

At block 1504, the core 102 writes a sync request to its sync register108 with a sync condition value of 10 (denoted sync 10 in FIG. 15). As aresult, the control unit 104 puts the core 102 to sleep. Flow proceedsto block 1506.

At block 1506, the core 102 gets awakened by the control unit 104 whenall enabled cores 102 have written a sync 10. It is noted that in thecase of a multi-die 406 microprocessor 100 configuration, the synccondition occurrence may be a multi-die sync condition occurrence. Thatis, the control unit 104 will wait to wakeup (or interrupt in the casewhere the cores 102 have not elected to sleep) the cores 102 until allthe cores 102 specified in the core set field 228 (which may includecores 102 on both dies 406) and which are enabled (as indicated by theenabled bits 254) have written their sync request. Flow proceeds todecision block 1508.

At decision block 1508, the core 102 determines whether it is the core102 that was instructed at block 1502 to disable itself. If so, flowproceeds to block 1517; otherwise, flow proceeds to block 1518.

At block 1517, the core 102 writes the disable core bit 236 to causeitself to be removed from the list of enabled cores 102, e.g., to haveits corresponding enabled bit 254 in the configuration word 252 of theconfiguration register 112 cleared. Afterward, the core 102 preventsitself from executing any more instructions, preferably by setting oneor more bits to turn off its clock signals and have its power removed.Flow ends at block 1517.

At block 1518, the core 102 writes a sync request to its sync register108 with a sync condition value of 11 (denoted sync 11 in FIG. 15). As aresult, the control unit 104 puts the core 102 to sleep. Flow proceedsto block 1522.

At block 1522, the core 102 gets awakened by the control unit 104 whenall enabled cores 102 have written a sync 11. Again, in the case of amulti-die 406 microprocessor 100 configuration, the sync conditionoccurrence may be a multi-die sync condition occurrence based on theupdated values in the configuration register 112. Furthermore, thecontrol unit 104 will exclude from consideration the core 102 thatdisabled itself at block 1517 as it determines whether a sync conditionoccurred. More specifically, in the circumstances in which all the othercores 102 (other than the core 102 disabling itself) write a sync 11before the disabling-itself core 102 writes the sync register 108 atblock 1517, then the control unit 104 will detect the sync conditionoccurred (at block 316) when the disabling-itself core 102 writes thesync register 108 at block 1517 because at that point the control unit104 no longer considers the disabled core 102 when determining whetherthe sync condition has occurred because the enabled bit 254 of thedisabled core 102 is clear (see FIG. 16). That is, the control unit 104determines that the sync condition has occurred because all of theenabled cores 102 have written a sync 11, regardless of whether thedisabled core 102 has written a sync 11. Flow proceeds to block 1524.

At block 1524, the core 102 reads the configuration register 112, whoseconfiguration word 252 will reflect the disabled core 102 that wasdisabled at block 1517. The core 102 then generates itsconfiguration-related values based on the new value of the configurationword 252. Preferably, the disable instruction of block 1502 is executedby system firmware (e.g., BIOS setup) and, after the core 102 isdisabled, the system firmware performs a reboot of the system, e.g.,after block 1526. During the reboot, the microprocessor 100 may operatedifferently than before the generation of the configuration-relatedvalues here at block 1524. For example, the BSP during the reboot may bea different core 102 than before the generation of theconfiguration-related values. For another example, the systemconfiguration information (e.g., the number of cores 102 or logicalprocessors in the system) determined by the BSP prior to bootstrappingthe operating system and saved in memory for the operating system toread it after it is booted may be different. For another example, theAPIC IDs of the cores 102 still enabled may be different than before thegeneration of the configuration-related values, in which case theoperating system will direct interrupt requests and the cores 102 willrespond to the interrupt requests differently than before the generationof the configuration-related values. For another example, the mastercore 102 of blocks 907 and 919 that performs the package sleep statehandshake protocol of FIG. 9 may be a different core 102 than before thegeneration of the configuration-related values. Flow proceeds todecision block 1526.

At block 1526, the core 102 resumes the task it was performing before itwas interrupted at block 1532. Flow ends at block 1526.

The embodiments for dynamically reconfiguring the microprocessor 100described herein may be used in a variety of applications. For example,the dynamic reconfiguration may be used for testing and/or simulationduring development of the microprocessor 100 and/or for field-testing.Also, a user may want to know the performance and/or amount of powerconsumed when running a given application using only a subset of thecores 102. In one embodiment, after a core 102 is disabled, it may haveits clocks turned off and/or power removed such that it consumesessentially no power. Furthermore, in a high reliability system, eachcore 102 may periodically check if the other cores 102 are faulty and ifthe cores 102 vote that a given core 102 is faulty, the healthy coresmay disable the faulty core 102 and cause the remaining cores 102 toperform a dynamic reconfiguration such as described. In such anembodiment, the control word 202 may include an additional field thatenables the writing core 102 to specify the core 102 to be disabled andthe operation described with respect to FIG. 15 is modified such that acore may disable a different core 102 than itself at block 1517.

Referring now to FIG. 16, a timing diagram illustrating an example ofthe operation of the microprocessor 100 according to the flowchart ofFIG. 15 is shown. In the example, a configuration of a microprocessor100 with three cores 102, denoted core 0, core 1 and core 2, is shown;however, it should be understood that in other embodiments themicroprocessor 100 may include different numbers of cores 102 and may bea single-die or multi-die microprocessor 100. In the timing diagram, thetiming of events proceeds downward.

Core 1 encounters an instruction to disable itself and in response sendsa reconfigure message and interrupt to core 0 and core 2 (per block1502). Core 1 then writes a sync 10 and is put to sleep (per block1504).

Each of core 0 and core 2 eventually are interrupted from their currenttasks and read the message (per block 1532). In response, each of core 0and core 2 writes a sync 10 and is put to sleep (per block 1504). Asshown, the time at which each of the cores writes the sync 10 may vary,for example due to the latency of the instruction that is executing whenthe interrupt is asserted.

When all the cores have written the sync 10, the control unit 104 wakesthem all up simultaneously (per block 1506). Cores 0 and 2 thendetermine that they are not disabling themselves (per decision block1508) and write a sync 11 and are put to sleep (per block 1518).However, core 1 determines that it is disabling itself, so it writes itsdisable core bit 236 (per block 1517). In the example, core 1 writes itsdisable core bit 236 after cores 1 and 2 write their sync 11, as shown.Nevertheless, the control unit 104 detects the sync condition occurrencebecause the control unit 104 determines that the S bit 222 is set forevery core 102 whose enabled bit 254 is set. That is, even though the Sbit 222 of core 1 is not set, its enabled bit 254 was cleared at thewrite of the sync register 108 of core 1 at block 1517.

When all the enabled cores have written the sync 11, the control unit104 wakes them all up simultaneously (per block 1522). As describedabove, in the case of a multi-die microprocessor 100, when core 1 writesits disable core bit 236 and the local control unit 104 responsivelyclears the local enabled bit 254 of core 1, the local control unit 104also propagates the local enabled bits 254 to the remote die 406.Consequently, the remote control unit 104 also detects the synccondition occurrence and simultaneously wakes up all the enabled coresof its die 406. Cores 0 and 2 then generate their configuration-relatedvalues (per block 1524) based on the updated configuration register 112value and resume their pre-interrupt activity (per block 1526).

Hardware Semaphore

Referring now to FIG. 17, a block diagram illustrating the hardwaresemaphore 118 of FIG. 1 is shown. The hardware semaphore 118 includes anowned bit 1702, owner bits 1704 and a state machine 1706 that updatesthe owned bit 1702 and the owner bits 1704 in response to reads andwrites of the hardware semaphore 118 by the cores 102. Preferably, thenumber of owner bits 1704 is log₂ of the number of cores 102 of themicroprocessor 100 configuration in order to uniquely identify whichcore 102 currently owns the hardware semaphore 118. In anotherembodiment, the owner bits 1704 include one respective bit per core 102of the microprocessor 100. It is noted that although one set of ownedbit 1702, owner bits 1704 and state machine 1706 are described thatimplement a single hardware semaphore 118, the microprocessor 100 mayinclude a plurality of hardware semaphores 118 each including the set ofhardware described. Preferably, the microcode running on each of thecores 102 reads and writes the hardware semaphores 118 to gain ownershipof a resource that is shared by the cores 102, examples of which aredescribed below, in order to perform operations that require exclusiveaccess to the shared resource. The microcode may associate each one ofthe multiple hardware semaphores 118 with ownership of a differentshared resource of the microprocessor 100. Preferably, the hardwaresemaphore 118 is readable and writeable by the cores 102 at apredetermined address within a non-architectural address space of thecores 102. The non-architectural address space can only be accessed bymicrocode of a core 102, but cannot be accessed directly by userprograms (e.g., x86 architecture program instructions). Operation of thestate machine 1706 to update the owned bit 1702 and owner bits 1704 ofthe hardware semaphore 118 are described below with respect to FIGS. 18and 19, and uses of the hardware semaphore 118 are described thereafter.

Referring now to FIG. 18, a flowchart illustrating operation of thehardware semaphore 118 when read by a core 102 is shown. Flow begins atblock 1802.

At block 1802, a core 102, denoted core x, reads the hardware semaphore118. As described above, preferably the microcode of the core 102 readsthe predetermined address at which the hardware semaphore 118 resideswithin the non-architectural address space. Flow proceeds to decisionblock 1804.

At decision block 1804, the state machine 1706 examines the owner bits1704 to determine whether core x is the owner of the hardware semaphore118. If so, flow proceeds to block 1808; otherwise, flow proceeds toblock 1806.

At block 1806, the hardware semaphore 118 returns to the reading core102 a zero value to indicate that the core 102 does not own the hardwaresemaphore 118. Flow ends at block 1806.

At block 1808, the hardware semaphore 118 returns to the reading core102 a one value to indicate that the core 102 owns the hardwaresemaphore 118. Flow ends at block 1808.

As described above, the microprocessor 100 may include a plurality ofhardware semaphores 118. In one embodiment, the microprocessor 100includes 16 hardware semaphores 118, and when a core 102 reads thepredetermined address it receives a 16-bit data value in which each bitcorresponds to a different one of the 16 hardware semaphores 118 andindicates whether or not the core 102 reading the predetermined addressowns the corresponding hardware semaphore 118.

Referring now to FIG. 19, a flowchart illustrating operation of thehardware semaphore 118 when written by a core 102 is shown. Flow beginsat block 1902.

At block 1902, a core 102, denoted core x, writes the hardware semaphore118, e.g., at the non-architectural predetermined address describedabove. Flow proceeds to decision block 1904.

At decision block 1904, the state machine 1706 examines the owned bit1702 to determine whether the hardware semaphore 118 is owned by any ofthe cores 102 or is free, i.e., un-owned. If owned, flow proceeds todecision block 1914; otherwise, flow proceeds to decision block 1906.

At decision block 1906, the state machine 1706 examines the valuewritten. If the value is one, which indicates the core 102 would like toobtain ownership of the hardware semaphore 118, flow proceeds to block1908; whereas, if the value is zero, which indicates the core 102 wouldlike to relinquish ownership of the hardware semaphore 118, flowproceeds to block 1912.

At block 1908, the state machine 1706 updates the owned bit 1702 to aone and sets the owner bits 1704 to a value indicating core x now ownsthe hardware semaphore 118. Flow ends at block 1908.

At block 1912, the state machine 1706 performs no update of the ownedbit 1702 nor the owner bits 1704. Flow ends at block 1912.

At decision block 1914, the state machine 1706 examines the owner bits1704 to determine whether core x is the owner of the hardware semaphore118. If so, flow proceeds to decision block 1916; otherwise, flowproceeds to block 1912.

At decision block 1916, the state machine 1706 examines the valuewritten. If the value is one, which indicates the core 102 would like toobtain ownership of the hardware semaphore 118, flow proceeds to block1912 (where no update occurs, since this core 102 already owns thehardware semaphore 118, as determined at decision block 1914); whereas,if the value is zero, which indicates the core 102 would like torelinquish ownership of the hardware semaphore 118, flow proceeds toblock 1918.

At block 1918, the state machine 1706 updates the owned bit 1702 to azero to indicate that now no core 102 owns the hardware semaphore 118.Flow ends at block 1918.

As described above, in one embodiment the microprocessor 100 includes 16hardware semaphores 118. When a core 102 writes the predeterminedaddress it writes a 16-bit data value in which each bit corresponds to adifferent one of the 16 hardware semaphores 118 and indicates whetherthe core 102 writing the predetermined address is requesting to own (onevalue) or to relinquish ownership (zero value) of the correspondinghardware semaphore 118.

In one embodiment, arbitration logic arbitrates requests by the cores102 to access the hardware semaphore 118 such that reads/writes from/tothe hardware semaphore 118 are serialized. In one embodiment, thearbitration logic employs a round-robin fairness algorithm among thecores 102 for access to the hardware semaphore 118.

Referring now to FIG. 20, a flowchart illustrating operation of themicroprocessor 100 to employ the hardware semaphore 118 to perform anaction that requires exclusive ownership of a resource is shown. Morespecifically, the hardware semaphore 118 is used to insure that only onecore 102 at a time performs a write back and invalidate of the sharedcache memory 119 in the situation where two or more of the cores 102have each encountered an instruction to write back and invalidate theshared cache 119. The operation is described from the perspective of asingle core, but each of the cores 102 of the microprocessor 100operates according to the description to collectively insure that whileone core 102 is performing a write back and invalidate operation othercores 102 are not. That is, the operation of FIG. 20 insures that WBINVDinstruction processes are serialized. In one embodiment, the operationof FIG. 20 may be performed in a microprocessor 100 that performs aWBINVD instruction according to the embodiment of FIG. 7. Flow begins atblock 2002.

At block 2002, a core 102 encounters a cache control instruction, suchas a WBINVD instruction. Flow proceeds to block 2004.

At block 2004, the core 102 writes a one to the WBINVD hardwaresemaphore 118. In one embodiment, the microcode has allocated one of thehardware semaphores 118 to the WBINVD operation. The core 102 then readsthe WBINVD hardware semaphore 118 to determine whether it obtainedownership. Flow proceeds to decision block 2006.

At decision block 2006, if the core 102 determines that it obtainedownership of the WBINVD hardware semaphore 118, flow proceeds to block2008; otherwise, flow returns to block 2004 to attempt again to obtainthe ownership. It is noted that as microcode of the instant core 102loops through blocks 2004 and 2006, it will eventually be interrupted bythe core 102 that owns the WBINVD hardware semaphore 118, since thatcore 102 is performing a WBINVD instruction and sends the instant core102 an interrupt at block 702 of FIG. 7. Preferably, each time throughthe loop, the instant core 102 microcode checks an interrupt statusregister to see whether one of the other cores 102 (e.g., the core 102that owns the WBINVD hardware semaphore 118) sent an interrupt to theinstant core 102. The instant core 102 will then perform the operationsof FIG. 7 and at block 749 will resume operation according to FIG. 20 toattempt to gain ownership of the hardware semaphore 118 to perform itsWBINVD instruction.

At block 2008, the core 102 has obtained ownership and flow proceeds toblock 702 of FIG. 7 to perform the WBINVD instruction. As part of theWBINVD instruction operation, at block 748 of FIG. 7 the core 102 writesa zero to the WBINVD hardware semaphore 118 to relinquish ownership ofit. Flow ends at block 2008.

An operation similar to the operation described with respect to FIG. 20may be employed by the microcode in order to obtain exclusive ownershipof other shared resources. Other resources to which a core 102 mayobtain exclusive ownership by using a hardware semaphore 118 are uncore103 registers that are shared by the cores 102. In one embodiment, theuncore 103 register comprises a control register that includes arespective field for each of the cores 102. The field controls anoperational aspect of the respective core 102. Because the fields are inthe same register, when a core 102 wants to update its respective fieldbut not the fields of the other cores 102, the core 102 must read thecontrol register, modify the value read, and then write back themodified value to the control register. For example, the microprocessor100 may include an uncore 103 performance control register (PCR) thatcontrols the bus clock ratio of the cores 102. To update its bus clockratio, a given core 102 must read, modify, and write back the PCR.Therefore, in one embodiment the microcode is configured to perform aneffectively atomic read/modify/write of the PCR by doing so only if thecore 102 owns the hardware semaphore 118 associated with the PCR. Thebus clock ratio determines the individual core 102 clock frequency as amultiple of the frequency of the clock supplied to the microprocessor100 via an external bus.

Another resource is a Trusted Platform Module (TPM). In one embodiment,the microprocessor 100 implements a TPM in microcode that runs on thecores 102. At a given instant in time, the microcode running on one andonly one of the cores 102 of the microprocessor 100 is implementing theTPM; however, the core 102 implementing the TPM may change over time. Byusing the hardware semaphore 118 associated with the TPM, the microcodeof the cores 102 assures that only one core 102 is implementing the TPMat a time. More specifically, the core 102 currently implementing theTPM writes the TPM state to the PRAM 116 prior to giving upimplementation of the TPM and the core 102 that takes overimplementation of the TPM reads the TPM state from the PRAM 116. Themicrocode in each of the cores 102 is configured such that when a core102 wants to become the core 102 implementing the TPM, the core 102first obtains ownership of the TPM hardware semaphore 118 before readingthe TPM state from the PRAM 116 and begins implementing the TPM. In oneembodiment, the TPM conforms substantially to a TPM specificationpublished by the Trusted Computing Group, such as the ISO/IEC 11889specification.

As described above, a conventional solution to resource contention amongmultiple processors is to employ a software semaphore in system memory.Potential advantages of the hardware semaphore 118 described herein arethat it may avoid generation of additional traffic on the externalmemory bus and it may be faster than accessing system memory.

Interrupting, Non-Sleeping Sync Requests

Referring now to FIG. 21, a timing diagram illustrating an example ofthe operation of the microprocessor 100 according to the flowchart ofFIG. 3 in which the cores 102 issue non-sleeping sync requests is shown.In the example, a configuration of a microprocessor 100 with three cores102, denoted core 0, core 1 and core 2, is shown; however, it should beunderstood that in other embodiments the microprocessor 100 may includedifferent numbers of cores 102.

Core 0 writes a sync 14 in which neither the sleep bit 212 nor the selwake bit 214 is set (i.e., a non-sleeping sync request). Consequently,the control unit 104 allows core 0 to keep running (per the “NO” branchout of decision block 312).

Core 1 also eventually writes a non-sleeping sync 14 and the controlunit 104 allows core 1 to keep running Finally, Core 2 writes anon-sleeping sync 14. As shown, the time at which each of the coreswrites the sync 14 may vary.

When all the cores have written the non-sleeping sync 14, the controlunit 104 simultaneously sends each of core 0, core 1 and core 2 a syncinterrupt (per block 334). Each core then receives the sync interruptand services it (unless the sync interrupt was masked, in which case themicrocode typically polls for it).

Designation of the Bootstrap Processor

In one embodiment, as described above, normally (e.g., when the “allcores BSP” feature of FIG. 23 is disabled) one core 102 designatesitself the bootstrap processor (BSP) and performs special duties, suchas bootstrapping the operating system. In one embodiment, normally(e.g., when the “modify BSP” and “all cores BSP” features of FIGS. 22and 23, respectively, are disabled) virtual core number zero is bydefault the BSP core 102.

However, the present inventors have observed that there may besituations where it is advantageous for the BSP to be designated in adifferent manner, embodiments of which are described below. For example,much of the testing of a microprocessor 100 part, particularly duringmanufacturing testing, is performed by booting the operating system andrunning programs to insure that the part 100 is working properly.Because the BSP core 102 performs the system initialization and bootsthe operating system, it may be exercised in ways that the AP cores 102may not. Additionally, it has been observed that, even in amultithreaded operating environment, the BSP typically bears a largershare of the processing burden than the APs; therefore, the AP cores 102may not be tested as thoroughly as the BSP core 102. Finally, there maybe certain actions that only the BSP core 102 performs on behalf of themicroprocessor 100 as a whole, such as the package sleep state handshakeprotocol as described with respect to FIG. 9.

Therefore, embodiments are described in which any of the cores 102 maybe designated the BSP. In one embodiment, during the testing of themicroprocessor 100, the tests are run N times, wherein N is the numberof cores 102 of the microprocessor 100, and in each run of the tests themicroprocessor 100 is re-configured to make the BSP a different core102. This may advantageously provide better test coverage duringmanufacturing and may also advantageously expose bugs in themicroprocessor 100 during its design process. Another advantage isduring different runs each core 102 may have a different APIC ID andconsequently respond to different interrupt requests, which may providemore extensive test coverage.

Referring now to FIG. 22, a flowchart illustrating a process forconfiguring the microprocessor 100 is shown. In the description of FIG.22 reference is made to the multi-die microprocessor 100 of FIG. 4,which includes two die 406 and eight cores 102. However, it should beunderstood that the dynamic reconfiguration described may apply to amicroprocessor 100 with a different configuration, namely with more thantwo dies or a single die, and more or less than eight cores 102 but atleast two cores 102. The operation is described from the perspective ofa single core, but each of the cores 102 of the microprocessor 100operates according to the description to collectively dynamicallyreconfigure the microprocessor 100. Flow begins at block 2202.

At block 2202, the microprocessor 100 is reset and performs an initialportion of its initialization, preferably in a manner similar to thatdescribed above with respect to FIG. 14. However, the generating of theconfiguration-related values, such as a block 1424 of FIG. 14, inparticular the APIC ID and BSP flag, is performed in the mannerdescribed here with respect to blocks 2203 through 2214. Flow proceedsto block 2203.

At block 2203, the core 102 generates its virtual core number,preferably as described above with respect to FIG. 14. Flow proceeds todecision block 2204.

At decision block 2204, the core 102 samples an indicator to determinewhether or not a feature is enabled. The feature is referred to hereinas the “modify BSP” feature. In one embodiment, blowing a fuse 114enables the modify BSP feature. Preferably, during testing, rather thanblowing the modify BSP feature fuse 114, a true value is scanned intothe holding register bit associated with the modify BSP feature fuse114, as described above with respect to FIG. 1, to enable the modify BSPfeature. In this way, the modify BSP feature is not permanently enabledon the microprocessor 100 part, but is instead disabled on subsequentpower-ups. Preferably, the operations at blocks 2203 through 2214 areperformed by microcode of the core 102. If the modify BSP feature isenabled, flow proceeds to block 2205. Otherwise, flow proceeds to block2206.

At block 2205, the core 102 modifies the virtual core number that wasgenerated at block 2203. In one embodiment, the core 102 modifies thevirtual core number to be the result of a rotate function of the virtualcore number generated at block 2203 and a rotate amount, as shown here:

virtual core number=rotate(rotate amount, virtual core number).

The rotate function, in one embodiment, rotates the virtual core numberamong the cores 102 by the rotate amount. The rotate amount is a valuethat is blown into a fuse 114, or preferably, is scanned into a holdingregister during testing. Table 1 shows the virtual core number for eachcore 102 whose ordered pair (die number 258, local core number 256) isshown in the left-hand column for each rotate amount shown in the toprow in an example configuration in which the number of dies 406 is twoand the number of cores 102 per die 406 is four and all cores 102 areenabled. In this fashion, the tester is empowered to cause the cores 102to generate their virtual core number, and consequently APIC ID, as anyvalid value. Although one embodiment for modifying the virtual corenumber is described, other embodiments are contemplated. For example,the rotate direction may be opposite that shown in Table 1. Flowproceeds to block 2206.

TABLE 1 0 1 2 3 4 5 6 7 (0, 0) 0 7 6 5 4 3 2 1 (0, 1) 1 0 7 6 5 4 3 2(0, 2) 2 1 0 7 6 5 4 3 (0, 3) 3 2 1 0 7 6 5 4 (1, 0) 4 3 2 1 0 7 6 5(1, 1) 5 4 3 2 1 0 7 6 (1, 2) 6 5 4 3 2 1 0 7 (1, 3) 7 6 5 4 3 2 1 0

At block 2206, the core 102 populates the local APIC ID register witheither the default virtual core number generated at block 2203 or themodified value generated at block 2205. In one embodiment, the APIC IDregister may be read by the core 102 from itself (e.g., by the BIOSand/or operating system) at memory address 0x0FEE00020; whereas, inanother embodiment, the APIC ID register may be read by the core 102 atMSR address 0x802. Flow proceeds to decision block 2208.

At decision block 2208, the core 102 determines whether its APIC IDpopulated at block 2208 is zero. If so, flow proceeds to block 2212;otherwise, flow proceeds to block 2214.

At block 2212, the core 102 sets its BSP flag to true to indicate thecore 102 is the BSP. In one embodiment, the BSP flag is a bit in the x86APIC base register (IA32_APIC_BASE MSR) of the core 102. Flow proceedsto decision block 2216.

At block 2214, the core 102 sets the BSP flag to false to indicate thecore 102 is not the BSP, i.e., in an AP. Flow proceeds to decision block2216.

At decision block 2216, the core 102 determines whether it is the BSP,i.e., whether it designated itself the BSP core 102 at block 2212 asopposed to designating itself an AP core 102 at block 2214. If the core102 is the BSP, flow proceeds to block 2218; otherwise, flow proceeds toblock 2222.

At block 2218, the core 102 begins fetching and executing the systeminitialization firmware (e.g., the BSP BIOS bootstrap code). This mayinclude instructions that implicate the BSP flag and the APIC ID, e.g.,instructions that read the APIC ID register or the APIC base register,in which case the core 102 returns the values written at blocks 2206 and2212/2214. This may also include being the only core 102 of themicroprocessor 100 to perform actions on behalf of the microprocessor100 as a whole, such as the package sleep state handshake protocol asdescribed with respect to FIG. 9. Preferably, the BSP core 102 beginsfetching and executing the system initialization firmware at anarchitecturally-defined reset vector. For example, in the x86architecture, the reset vector is address 0xFFFFFFF0. Preferably,executing the system initialization firmware includes bootstrapping theoperating system, e.g., loading the operating system and transferringcontrol to it. Flow proceeds to block 2224.

At block 2222, the core 102 halts itself and waits for a startupsequence from the BSP to begin fetching and executing instructions. Inone embodiment, the startup sequence received from the BSP includes aninterrupt vector to AP system initialization firmware (e.g., the AP BIOScode). This may include instructions that implicate the BSP flag and theAPIC ID, in which case the core 102 returns the values written at blocks2206 and 2212/2214. Flow proceeds to block 2224.

At block 2224, the core 102, as it executes instructions, receivesinterrupt requests and responds to the interrupt requests based on itsAPIC ID written in its APIC ID register at block 2206. Flow ends atblock 2224.

As described above, according to one embodiment, the core 102 whosevirtual core number is zero is the BSP by default. However, the presentinventors have observed that there may be situations where it isadvantageous for all of the cores 102 to be designated the BSP,embodiments of which are described below. For example, themicroprocessor 100 developer may have invested a significant amount oftime and cost to develop a large body of tests that are designed to runon a single core in a single-threaded manner, and the developer wouldlike to use the single core tests to test the multi-core microprocessor100. For example, the tests may run under the old and well-known DOSoperating system in x86 real mode.

Running these tests on each core 102 could be accomplished in a serialfashion using the modify BSP feature described above with respect toFIG. 22 and/or by blowing fuses or scanning into a holding registermodified fuse values to disable all cores 102 but the one core 102 to betested. However, the present inventors have recognized that this wouldtake more time (e.g., approximately 4× in the case of a 4-coremicroprocessor 100) than running the tests concurrently on all the cores102. Furthermore, the time required to test each individualmicroprocessor 100 part is precious, particularly when manufacturinghundreds of thousands or more of the microprocessor 100 parts andparticularly when much of the testing is performed on very expensivetest equipment.

Additionally, it may be the case that a speed path in the logic of themicroprocessor 100 is more heavily stressed when running more than onecore 102 (or all the cores 102) at the same time because this generatesmore heat and/or draws more power. Running the tests in the serialfashion might not generate the additional stress and expose the speedpath.

Therefore, embodiments are described in which dynamically all of thecores 102 may be designated the BSP core 102 so that all of the cores102 may execute a test concurrently.

Referring now to FIG. 23, a flowchart illustrating a process forconfiguring the microprocessor 100 according to an alternate embodimentis shown. In the description of FIG. 23 reference is made to themulti-die microprocessor 100 of FIG. 4, which includes two die 406 andeight cores 102. However, it should be understood that the dynamicreconfiguration described may apply to a microprocessor 100 with adifferent configuration, namely with more than two dies or a single die,and more or less than eight cores 102 but at least two cores 102. Theoperation is described from the perspective of a single core, but eachof the cores 102 of the microprocessor 100 operates according to thedescription to collectively dynamically reconfigure the microprocessor100. Flow begins at block 2302.

At block 2302, the microprocessor 100 is reset and performs an initialportion of its initialization, preferably in a manner similar to thatdescribed above with respect to FIG. 14. However, the generating of theconfiguration-related values, such as a block 1424 of FIG. 14, inparticular the APIC ID and BSP flag, is performed in the mannerdescribed here with respect to blocks blocks 2304 through 2312. Flowproceeds to decision block 2304.

At decision block 2304, the core 102 detects a feature is enabled. Thefeature is referred to herein as the “all cores BSP” feature.Preferably, blowing a fuse 114 enables the all cores BSP feature.Preferably, during testing, rather than blowing the all cores BSPfeature fuse 114, a true value is scanned into the holding register bitassociated with the all cores BSP feature fuse 114, as described abovewith respect to FIG. 1, to enable the all cores BSP feature. In thisway, the all cores BSP feature is not permanently enabled on themicroprocessor 100 part, but is instead disabled on subsequentpower-ups. Preferably, the operations at blocks 2304 through 2312 areperformed by microcode of the core 102. If the all cores BSP feature isenabled, flow proceeds to block 2305. Otherwise, flow proceeds to block2203 of FIG. 22.

At block 2305, the core 102 sets its virtual core number to zero,regardless of the local core number 256 and die number 258 of the core102. Flow proceeds to block 2306.

At block 2306, the core 102 populates the local APIC ID register withthe zero value of the virtual core number set at block 2305. Flowproceeds to block 2312.

At block 2312, the core 102 sets its BSP flag to true to indicate thecore 102 is the BSP, regardless of the local core number 256 and dienumber 258 of the core 102. Flow proceeds to block 2315.

At block 2315, whenever a core 102 performs a memory access request, themicroprocessor 100 modifies upper address bits of the memory accessrequest address differently for each core 102 so that each core 102accesses its own unique memory space. That is, depending on the core 102making the memory access request, the microprocessor 100 modifies theupper address bits so the upper address bits have a unique value foreach core 102. In one embodiment, the microprocessor 100 modifies theupper address bits as specified by values blown into fuses 114. In analternate embodiment, the microprocessor 100 modifies the upper addressbits based on the local core number 256 and die number 258 of the core102. For example, in an embodiment in which the number of cores 102 inthe microprocessor 100 is four, the microprocessor 100 modifies theupper two bits of the memory address and generates a unique value on theupper two bits for each core 102. Effectively, the memory spaceaddressable by the microprocessor 100 is divided into N sub-spaces,where N is the number of cores 102. The test programs are developed suchthat they limit themselves to specifying addresses within the lowest ofthe N sub-spaces. For example, assume the microprocessor 100 is capableof addressing 64 GB of memory and the microprocessor 100 includes fourcores 102. The test is developed to only access the bottom 8 GB ofmemory. When core 0 executes an instruction that accesses memory addressA (in the lower 8 GB of memory), the microprocessor 100 generates anaddress on the memory bus of A (unmodified); when core 1 executes aninstruction that accesses the same memory address A, the microprocessor100 generates an address on the memory bus of A+8 GB; when core 2executes an instruction that accesses the same memory address A, themicroprocessor 100 generates an address on the memory bus of A+16 GB;and when core 3 executes an instruction that accesses the same memoryaddress A, the microprocessor 100 generates an address on the memory busof A+32 GB. In this fashion, advantageously, the cores 102 will not becolliding in their accesses to memory, which enables the tests toexecute correctly. Preferably, the single-threaded tests are executed ona stand-alone testing machine that is capable of testing themicroprocessor 100 in isolation. The microprocessor 100 developerdevelops test data to be provided by the testing machine to themicroprocessor 100 in response to a memory read request; conversely, thedeveloper develops result data that the testing machine compares to thedata written by the microprocessor 100 during a memory write access toinsure that the microprocessor 100 is writing the correct data. In oneembodiment, shared cache 119 (i.e., the highest level cache thatgenerates the addresses used in the external bus transactions) is theportion of the microprocessor 100 configured to modify the upper addressbits when the all cores BSP feature is enabled. Flow proceeds to block2318.

At block 2318, the core 102 begins fetching and executing the systeminitialization firmware (e.g., the BSP BIOS bootstrap code). This mayinclude instructions that implicate the BSP flag and the APIC ID, e.g.,instructions that read the APIC ID register or the APIC base register,in which case the core 102 returns the zero value written at block 2306.Preferably, the BSP core 102 begins fetching and executing the systeminitialization firmware at an architecturally-defined reset vector. Forexample, in the x86 architecture, the reset vector is address0xFFFFFFF0. Preferably, executing the system initialization firmwareincludes bootstrapping the operating system, e.g., loading the operatingsystem and transferring control to it. Flow proceeds to block 2324.

At block 2324, the core 102, as it executes instructions, receivesinterrupt requests and responds to the interrupt requests based on itsAPIC ID value of zero written in its APIC ID register at block 2306.Flow ends at block 2324.

Although an embodiment has been described with respect to FIG. 23 inwhich all the cores 102 are designated the BSP, other embodiments arecontemplated in which multiple but less than all of the cores 102 aredesignated the BSP.

Although embodiments have been described in the context of an x86-stylesystem that employs a local APIC per core 102 and in which there is arelationship between the APIC ID and the BSP designation, it should beunderstood that the designation of the bootstrap processor is notlimited to x86-style embodiments, but may be employed in systems withdifferent system architectures.

Propagation of Microcode Patches to Multiple Cores

As may be observed from the foregoing, there may be many criticalfunctions performed largely by microcode of a microprocessor, andparticularly, which require correct communication and coordinationbetween the microcode instances executing on the multiple cores of themicroprocessor. Due to complexity of the microcode, a significantprobability that bugs will exist in the microcode that require fixing.This may be accomplished via microcode patches in which new microcodeinstructions are substituted for old microcode instructions that causethe bug. That is, the microprocessor includes special hardware thatfacilitates the patching of microcode. Typically, it is desirable toapply the microcode patch to all the cores of the microprocessor.Conventionally, this has been performed by separately executing anarchitectural instruction on each of the cores to apply the patch.However, the convention approach may be problematic.

First, the patch may pertain to inter-core communication by instances ofthe microcode (e.g., core synchronization, hardware semaphore use) or tofeatures that require microcode inter-core communication (e.g.,trans-core debug requests, cache control operations or power management,or dynamic multi-core microprocessor configuration). The execution ofthe architectural patch application instruction separately on each coremay create a window of time in which the microcode patch is applied tosome cores but not to others (or a previous patch is applied to somecores and the new patch is applied to others). This may cause acommunication failure between the cores and incorrect operation of themicroprocessor. Other problems, foreseen and unforeseen, may also becreated if all of the cores of the microprocessor do not have the samemicrocode patch applied.

Second, the architecture of the microprocessor specifies many featuresthat may be supported by some instances of the microprocessor and not byothers. During operation, the microprocessor is capable of communicatingto system software which particular features it supports. For example,in the case of an x86 architecture microprocessor, the x86 CPUIDinstruction may be executed by system software to determine thesupported feature set. However, the feature set-determining instruction(e.g., CPUID) is executed separately on each core of the microprocessor.In some cases, a feature may be disabled because a bug existed at thetime the microprocessor was released. However, subsequently a microcodepatch may be developed that fixes the bug so that the feature may now beenabled after the patch is applied. However, if the patch is applied inthe conventional manner (i.e., applied separately to each core throughthe separate execution of the apply patch instruction on each core),different cores may indicate different feature sets at a given point intime depending upon whether or not the patch has been applied to them.This may be problematic, particularly if the system software (such asthe operating system, for example, to facilitate thread migrationbetween the cores) expects all the cores of the microprocessor to havethe same feature set. In particular, it has been observed that somesystem software only obtains the feature set of one core and assumes theother cores have the same feature set.

Third, the microcode instance of each core controls and/or communicateswith uncore resources that are shared by the cores (e.g., sync-relatedhardware, hardware semaphore, share PRAM, shared cache or serviceprocessing unit). Therefore, generally speaking, it may be problematicfor the microcode of two different cores to be controlling orcommunicating with an uncore resource simultaneously in two differentmanners due to the fact that one of the cores has a microcode patchapplied and the other does not (or the two cores have differentmicrocode patches).

Finally, the microcode patch hardware of the microprocessor may be suchthat applying the patch in the conventional manner could potentiallycause interference of the operation of a patch by one core by theapplication of a patch by another core, for example, if portions of thepatch hardware are shared among the cores.

Advantageously, embodiments for applying a microcode patch to amulti-core microprocessor in an atomic manner at the architecturalinstruction level to potentially solve such problems are describedherein. The application of the patch is atomic in at least two senses.First, the patch is applied to the entire microprocessor 100 in responseto the execution of an architectural instruction on a single core 102.That is, the embodiments do not require system software to execute anapply microcode patch instruction (described below) on each core 102.More specifically, the single core 102 that encounters the applymicrocode patch instruction sends messages to and interrupts the othercores 102 to invoke instances of the portion of their microcode thatapplies the patch, and all the microcode instances cooperate with oneanother such that the microcode patch is applied to the microcode patchhardware of each of the cores 102 and to shared patch hardware of themicroprocessor 100 while interrupts are disabled on all the cores 102.Second, the microcode instances running on all the cores 102 thatimplement the atomic patch application mechanism cooperate with oneanother such that they refrain from executing any architecturalinstructions (other than the one apply microcode patch instruction)after all the cores 102 of the microprocessor 100 have agreed to applythe patch and until all the cores 102 have done so. That is, none of thecores 102 executes an architectural instruction while any of the cores102 is applying the microcode patch. Furthermore, in a preferredembodiment, all the cores 102 reach the same place in the microcode thatperforms the patch application with interrupts disabled, and after thatthe cores 102 execute only microcode instructions that apply themicrocode patch until all cores 102 of the microprocessor 100 confirmthe patch has been applied. That is, none of the cores 102 executemicrocode instructions other than those that apply the microcode patchwhile any of the cores 102 of the microprocessor 100 are applying thepatch.

Referring now to FIG. 24, a block diagram illustrating a multicoremicroprocessor 100 according to an alternate embodiment is shown. Themicroprocessor 100 is similar in many respects to the microprocessor 100of FIG. 1. However, the microprocessor 100 of FIG. 24 also includes inits uncore 103 a service processing unit (SPU) 2423, a SPU start addressregister 2497, an uncore microcode read-only memory (ROM) 2425 and anuncore microcode patch random access memory (RAM) 2408. Additionally,each core 102 includes a core PRAM 2499, a patch content-addressablememory (CAM) 2439 and a core microcode ROM 2404.

Microcode comprises microcode instructions. The microcode instructionsare non-architectural instructions stored within one or more memories(e.g., the uncore microcode ROM 2425, uncore microcode patch RAM 2408and/or core microcode ROM 2404) of the microprocessor 100 that arefetched by a core 102 based on a fetch address stored in thenon-architectural micro-program counter (micro-PC) and used by the core102 to implement the instructions of the instruction set architecture ofthe microprocessor 100. Preferably, the microcode instructions aretranslated by a microtranslator into microinstructions that are executedby execution units of the core 102, or in an alternate embodiment themicrocode instructions are executed directly by the execution units (inwhich case the microcode instructions are microinstructions). That themicrocode instructions are non-architectural instructions means they arenot instructions of the instruction set architecture (ISA) of themicroprocessor 100 but are instead encoded according to an instructionset distinct from the architectural instruction set. Thenon-architectural micro-PC is not defined by the instruction setarchitecture of the microprocessor 100 and is distinct from thearchitecturally-defined program counter of the core 102. The microcodeis used to implement some or all of the instructions of the instructionset of the microprocessor's ISA as follows. In response to decoding amicrocode-implemented ISA instruction, the core 102 transfers control toa microcode routine associated with the ISA instruction. The microcoderoutine comprises microcode instructions. The execution units executethe microcode instructions, or, according to the preferred embodiment,the microcode instructions are further translated into microinstructionsthat are executed by the execution units. The results of the executionof the microcode instructions (or microinstructions from which themicrocode instructions are translated) by the execution units are theresults defined by the ISA instruction. Thus, the collective executionof the microcode routine associated with the ISA instruction (or of themicroinstructions translated from the microcode routine instructions) bythe execution units “implements” the ISA instruction; that is, thecollective execution by the execution units of the implementingmicrocode instructions (or of the microinstructions translated from themicrocode instructions) performs the operation specified by the ISAinstruction on inputs specified by the ISA instruction to produce aresult defined by the ISA instruction. Additionally, the microcodeinstructions may be executed (or translated into microinstructions thatare executed) when the microprocessor is reset in order to configure themicroprocessor.

The core microcode ROM 2404 holds microcode executed by the particularcore 102 that comprises it. The uncore microcode ROM 2425 also holdsmicrocode executed by the cores 102; however, in contrast to the coreROMs 2404, the uncore ROM 2425 is shared by the cores 102. Preferably,the uncore ROM 2425 holds microcode routines that require lessperformance and/or are less frequently executed, since the access timeof the uncore ROM 2425 is greater than the core ROM 2404. Additionally,the uncore ROM 2425 holds code that is fetched and executed by the SPU2423.

The uncore microcode patch RAM 2408 is also shared by the cores 102. Theuncore microcode patch RAM 2408 holds microcode instructions executed bythe cores 102. The patch CAM 2439 holds patch addresses output by thepatch CAM 2439 to a microsequencer in response to a microcode fetchaddress if the fetch address matches the contents of one of the entriesin the patch CAM 2439. In such case, the microsequencer outputs thepatch address as the microcode fetch address rather than the nextsequential fetch address (or target address in the case of a branchingtype instruction), in response to which the uncore patch RAM 2408outputs a patch microcode instruction. This effectuates fetching of apatch microcode instruction from the uncore patch RAM 2408, ratherfetching a microcode instruction from the uncore ROM 2425 or the coreROM 2404 that is undesirable, for example because it and/or microcodeinstructions following it are the source of a bug. Thus, the patchmicrocode instruction effectively replaces, or patches, the undesirablemicrocode instruction that resides in the core ROM 2404 or the uncoremicrocode ROM 2425 at the original microcode fetch address. Preferably,the patch CAM 2439 and patch RAM 2408 are loaded in response toarchitectural instructions included in system software, such as BIOS orthe operating system running on the microprocessor 100.

The uncore PRAM 116, among other things, is used by the microcode tostore values used by the microcode. Some of these values effectivelyfunction as constant values because they are immediate values stored inthe core microcode ROM 2404 or the uncore microcode ROM 2425 or blowninto the fuses 114 at the time the microprocessor 100 is manufacturedand are written to the uncore PRAM 116 by the microcode when themicroprocessor 100 is reset and are not modified during operation of themicroprocessor 100, except possibly via a patch or in response to theexecution of an instruction that explicitly modifies the value, such asa WRMSR instruction. Advantageously, these values may be modified viathe patch mechanism described herein without requiring a change to thecore microcode ROM 2404 or the uncore microcode ROM 2425, which would becostly, and without requiring one or more fuses 114 to be unblown.

Additionally, the uncore PRAM 116 is used to hold patch code fetched andexecuted by the SPU 2423, as described herein.

The core PRAM 2499, similar to the uncore PRAM 116, is private, ornon-architectural, in the sense that it is not in the architectural userprogram address space of the microprocessor 100. However, unlike theuncore PRAM 116, each of the core PRAM 2499 is accessed only by itsrespective core 102 and is not shared by the other cores 102. Like theuncore PRAM 116, the core PRAM 2499 is also used by the microcode tostore values used by the microcode. Advantageously, these values may bemodified via the patch mechanism described herein without requiring achange to the core microcode ROM 2404 or uncore microcode ROM 2425.

The SPU 2423 comprises a stored program processor that is n adjunct toand distinct from each of the cores 102. Although the cores 102 arearchitecturally visible to execute instructions of the ISA of the cores102 (e.g., x86 ISA instructions), the SPU 2423 is not architecturallyvisible to do so. So, for example, the operating system cannot run onthe SPU 2423 nor can the operating system schedule programs of the ISAof the cores 102 (e.g., x86 ISA instructions) to run on the SPU 2423.Stated alternatively, the SPU 2423 is not a system resource managed bythe operating system. Rather, the SPU 2423 performs operations used todebug the microprocessor 100. Additionally, the SPU 2423 may assist inmeasuring performance of the cores 102, as well as other functions.Preferably, the SPU 2423 is much smaller, less complex and less powerconsuming (e.g., in one embodiment, the SPU 2423 includes built-in clockgating) than the cores 102. In one embodiment, the SPU 2423 comprises aFORTH CPU core.

There are asynchronous events that can occur with which debug microcodeexecuted by the cores 102 (referred to as tracer) cannot deal well.However, advantageously, the SPU 2423 can be commanded by a core 102 todetect the events and to perform actions, such as creating a log ormodifying aspects of the behavior of the cores 102 and/or external businterface of the microprocessor 100, in response to detecting theevents. The SPU 2423 can provide the log information to the user, and itcan also interact with the tracer to request the tracer to provide thelog information or to request the tracer to perform other actions. Inone embodiment, the SPU 2423 has access to control registers of thememory subsystem and programmable interrupt controller of each core 102,as well as to control registers of the shared cache 119.

Examples of the events that SPU 2423 can detect include the following:(1) a core 102 is hung, i.e., the core 102 has not retired anyinstructions for a number of clock cycles that is programmable; (2) acore 102 loads data from an uncacheable region of memory; (3) a changein temperature of the microprocessor 100 occurs; (4) the operatingsystem requests a change in the microprocessor's 100 bus clock ratioand/or requests a change in the microprocessor's 100 voltage level; (5)the microprocessor 100, of its own accord, changes the voltage leveland/or bus clock ratio, e.g., to achieve power savings or performanceimprovement; (6) an internal timer of a core 102 expires; (7) a cachesnoop that hits a modified cache line causing the cache line to bewritten back to memory occurs; (8) the temperature, voltage, or busclock ratio of the microprocessor 100 goes outside a respective range;(9) an external trigger signal is asserted by a user on an external pinof the microprocessor 100.

Advantageously, because the SPU 2423 is running code 132 independentlyof the cores 102, it does not have the same limitations as the tracermicrocode that executes on the cores 102. Thus, the SPU 2423 can detector be notified of the events independent of the core 102 instructionexecution boundaries and without disrupting the state of the core 102.

The SPU 2423 has its own code that it executes. The SPU 2423 may fetchits code from either the uncore microcode ROM 2425 or from the uncorePRAM 116. That is, preferably, the SPU 2423 shares the uncore ROM 2425and uncore PRAM 116 with the microcode that runs on the cores 102. TheSPU 2423 uses the uncore PRAM 116 to store its data, including the log.In one embodiment, the SPU 2423 also includes its own serial portinterface through which it can transmit the log to an external device.Advantageously, the SPU 2423 can also instruct tracer running on a core102 to store the log information from the uncore PRAM 116 to systemmemory.

The SPU 2423 communicates with the cores 102 via status and controlregisters. The SPU status register includes a bit corresponding to eachof the events described above that the SPU 2423 can detect. To notifythe SPU 2423 of an event, a core 102 sets the bit in the SPU statusregister corresponding to that event. Some of the event bits are set byhardware of the microprocessor 100 and some are set by microcode of thecores 102. The SPU 2423 reads the status register to determine the listof events that have occurred. One of the control registers includes abit corresponding to each action that the SPU 2423 should take inresponse to detecting one of the events specified in the statusregister. That is, a set of actions bits exists in the control registerfor each possible event in the status register. In one embodiment, thereare 16 action bits per event. In one embodiment, when the statusregister is written to indicate an event, this causes the SPU 2423 to beinterrupted, in response to which the SPU 2423 reads the status registerto determine which events have occurred. Advantageously, this savespower by alleviating the need for the SPU 2423 to poll the statusregister. The status register and control registers can also be read andwritten by user programs that execute instructions, such as RDMSR andWRMSR instructions.

The set of actions the SPU 2423 can perform in response to detecting anevent include the following. (1) Write the log information to the uncorePRAM 116. For each of the log-writing actions, multiple of the actionbits exist to enable the programmer to specify that only particularsubsets of the log information should be written. (2) Write the loginformation from the uncore PRAM 116 to the serial port interface. (3)Write to one of the control registers to set an event for the tracer.That is, the SPU 2423 can interrupt a core 102 and cause the tracermicrocode to be invoked to perform a set of actions associated with theevent. The actions may be specified by the user beforehand. In oneembodiment, when the SPU 2423 writes the control register to set theevent, this causes the core 102 to take a machine check exception, andthe machine check exception handler checks to see whether the tracer isactivated. If so, the machine check exception handler transfers controlto the tracer. The tracer reads the control register and if the eventsset in the control register are events that the user has enabled for thetracer, the tracer performs the actions specified beforehand by the userassociated with the events. For example, the SPU 2423 can set an eventto cause the tracer to write the log information stored in the uncorePRAM 116 to system memory. (4) Write to a control register to cause themicrocode to branch to a microcode address specified by the SPU 2423.This is particularly useful if the microcode is in an infinite loop suchthat the tracer will not be able to perform any meaningful actions, yetthe core 102 is still executing and retiring instructions, which meansthe processor hung event will not occur. (5) Write to a control registerto cause a core 102 to reset. As mentioned above, the SPU 2423 candetect that a core 102 is hung (i.e., has not retired any instructionfor some programmable amount of time) and reset it. The reset microcodechecks to see whether the reset was initiated by the SPU 2423 and, ifso, advantageously writes the log information out to system memorybefore clearing it during the process of initializing the core 102. (6)Continuously log events. In this mode, rather than waiting to beinterrupted about an event, the SPU 2423 spins in a loop checking thestatus register and continuously logging information to the uncore PRAM116 associated with the events indicated therein, and optionallyadditionally writing the log information to the serial port interface.(7) Write to a control register to stop a core 102 from issuing requeststo the shared cache 119 and/or stop the shared cache 119 fromacknowledging requests to the cores 102. This may be particularly usefulin debugging memory subsystem-related design bugs, such as pagetablewalk hardware bugs, and even fixing the bugs during operation ofthe microprocessor 100, such as through a patch to the SPU 2423 code, asdescribed below. (8) Write to a control register of an external businterface controller of the microprocessor 100 to perform transactionson the eternal system bus, such as special cycles or memory read/writecycles. (9) Write to a control register of the programmable interruptcontroller of a core 102 to generate an interrupt to another core 102 orto emulate an I/O device to the cores 102 or to fix a bug in theinterrupt controller, for example. (10) Write to a control register ofthe shared cache 119 to control its sizing, e.g., to disable or enabledifferent ways of the associative cache 119. (11) Write to controlregisters of the various functional units of the cores 102 to configuredifferent performance features, such as branch prediction or dataprefetch algorithms. As described below, advantageously the SPU 2423code may be patched, which enables the SPU 2423 to perform actions suchas those described herein to remedy design flaws or perform otherfunctions even after the design of the microprocessor 100 has beencompleted and the microprocessor 100 has been fabricated.

The SPU start address register 2497 holds the address at which the SPU2423 begins fetching instructions when it comes out of reset. The SPUstart address register is written by the cores 102. The address may bein either uncore PRAM 116 or uncore microcode ROM 2425.

Referring now to FIG. 25, a block diagram illustrating the structure ofa microcode patch 2500 according to one embodiment is shown. In theembodiment of FIG. 25, the microcode patch 2500 includes the followingportions: a header 2502; an immediate patch 2504; a checksum 2506 of theimmediate patch 2504; CAM data 2508; a core PRAM patch 2512; a checksumof the CAM data 2508 and core PRAM patch 2512; a RAM patch 2516; anuncore PRAM patch 2518; and a checksum 2522 of the core PRAM patch 2512and RAM patch 2516. The checksums 2506/2514/2522 enable themicroprocessor 100 to verify the integrity of the respective portions ofthe patches after they are loaded into the microprocessor 100.Preferably, the portions of the microcode patch 2500 are loaded fromsystem memory and/or from a non-volatile system, such as from a ROM orFLASH memory that holds a system BIOS or extensible firmware, forexample. The header 2502 describes each portion of the patch 2500, suchas its size, the location in its respective patch-related memory towhich the patch portion is to be loaded, and a valid flag that indicateswhether or not the portion contains a valid patch that should be appliedto the microprocessor 100.

The immediate patch 2504 comprises code (i.e., instructions, preferablymicrocode instructions) to be loaded into the uncore microcode patch RAM2408 of FIG. 24 (e.g., at block 2612 of FIG. 26) and then executed byeach of the cores 102 (e.g., at block 2616 of FIG. 26). The patch 2500also specifies the address to which the immediate patch 2504 is to beloaded into the patch RAM 2408. Preferably, the immediate patch 2504code modifies default values that were written by the reset microcode,such as values written to configuration registers that affect theconfiguration of the microprocessor 100. After the immediate patch 2504is executed by each of the cores 102 out of the patch RAM 2408, it isnot executed again. Furthermore, the subsequent loading of the RAM patch2516 into the patch RAM 2408 (e.g., at block 2632 of FIG. 26) mayoverwrite the immediate patch 2504 in the patch RAM 2408.

The RAM patch 2516 comprises the patch microcode instructions to beexecuted in place of the microcode instructions in the core ROM 2404 oruncore ROM 2425 that need to be patched. The RAM patch 2516 alsoincludes the address of the location in the patch RAM 2408 into whichthe patch microcode instructions are to be written when the patch 2500is applied (e.g., at block 2632 of FIG. 26). The CAM data 2508 is loadedinto the patch CAM 2439 of each core 102 (e.g., at block 2626 of FIG.26). As described above with respect to the operation of the patch CAM2439, the CAM data 2508 includes one or more entries each of whichcomprises a pair of microcode fetch addresses. The first address is ofthe microcode instruction to be patched and is the content matched bythe fetch address. The second address points to the location in thepatch RAM 2408 that holds the patch microcode instruction to be executedin place of the microcode instruction to be patched. Unlike theimmediate patch 2504, the RAM patch 2516 remains in the patch RAM 2408and (along with the operation of the patch CAM 2439 according to the CAMdata 2508) continues to function to patch the microcode of the coremicrocode ROM 2404 and/or the uncore microcode ROM 2425 until modifiedby another patch 2500 or the microprocessor 100 is reset.

The core PRAM patch 2512 includes data to be written to the core PRAM2499 of each core 102 and the address within the core PRAM 2499 to whicheach item of the data is to be written (e.g., at block 2626 of FIG. 26).The uncore PRAM patch 2518 includes data to be written to the uncorePRAM 116 and the address within the uncore PRAM 116 to which each itemof the data is to be written (e.g., at block 2632 of FIG. 26).

Referring now to FIG. 26, a flowchart illustrating operation of themicroprocessor 100 of FIG. 24 to propagate a microcode patch 2500 ofFIG. 25 to multiple cores 102 of the microprocessor 100 is shown. Theoperation is described from the perspective of a single core, but eachof the cores 102 of the microprocessor 100 operates according to thedescription to collectively propagate the microcode patch to all of thecores 102 of the microprocessor 100. More specifically, FIG. 26describes the operation of one core that encounters the instruction toapply a patch to the microcode, whose flow begins at block 2602, and theoperation of the other cores 102, whose flow begins at block 2652. Itshould be understood that multiple patches 2500 might be applied to themicroprocessor 100 at different times during operation of themicroprocessor 100. For example, a first patch 2500 may be appliedaccording to the atomic embodiments described herein when the systemthat includes the microprocessor 100 is bootstrapped, such as duringBIOS initialization, and a second patch 2500 may be applied after theoperating system is running, which may be particularly useful forpurposes of debugging the microprocessor 100.

At block 2602, one of the cores 102 encounters an instruction thatinstructs it to apply a microcode patch into the microprocessor 100.Preferably, the microcode patch is similar to that described above. Inone embodiment, the apply microcode patch instruction is an x86 WRMSRinstruction. In response to the apply microcode patch instruction, thecore 102 disables interrupts and traps to microcode that implements theapply microcode patch instruction. It should be understand that thesystem software that includes the apply microcode patch instruction mayinclude a sequence of multiple instructions to prepare for theapplication of the microcode patch; however, preferably, it is inresponse to a single architectural instruction of the sequence that themicrocode patch is propagated to all of the cores 102 in an atomicfashion at the architectural instruction level. That is, once interruptsare disabled on the first core 102 (i.e., the core 102 that encountersthe apply microcode patch instruction at block 2602), interrupts remaindisabled while the implementing microcode propagates the microcode patchand it is applied to all the cores 102 of the microprocessor 100 (e.g.,until after block 2634); furthermore, once interrupts are disabled onthe other cores 102 (e.g., at block 2652), they remain disabled untilthe microcode patch has been applied to all the cores 102 of themicroprocessor 100 (e.g., until after block 2634). Thus, advantageously,the microcode patch is propagated and applied to all of the cores 102 ofthe microprocessor 100 in an atomic fashion at the architecturalinstruction level. Flow proceeds to block 2604.

At block 2604, the core 102 obtains ownership of the hardware semaphore118 of FIG. 1. Preferably, the microprocessor 100 includes a hardwaresemaphore 118 associated with patching microcode. Preferably, the core102 obtains ownership of the hardware semaphore 118 in a manner similarto that described above with respect to FIG. 20, and more particularlywith respect to block 2004 and 2006. The hardware semaphore 118 is usedbecause it is possible while one of the cores 102 is applying a patch2500 in response to encountering an apply microcode patch instruction, asecond core 102 encounters an apply microcode patch instruction, inresponse to which the second core would begin to apply the second patch2500, which might result in incorrect execution, for example, due tocorruption of the first patch 2500. Flow proceeds to block 2606.

At block 2606, the core 102 sends a patch message to the other cores 102and sends them an inter-core interrupt. Preferably, the core 102 trapsto microcode in response to the apply microcode patch instruction (atblock 2602) or in response to the interrupt (at block 2652) and remainsin microcode, during which time interrupts are disabled (i.e., themicrocode does not allow itself to be interrupted), until block 2634.Flow proceeds from block 2606 to block 2608.

At block 2652, one of the other cores 102 (i.e., a core 102 other thanthe core 102 that encountered the apply microcode patch instruction atblock 2602) gets interrupted and receives the patch message as a resultof the inter-core interrupt sent at block 2606. In one embodiment, thecore 102 takes the interrupt at the next architectural instructionboundary (e.g., at the next x86 instruction boundary). In response tothe interrupt, the core 102 disables interrupts and traps to microcodethat handles the patch message. As described above, although flow atblock 2652 is described from the perspective of a single core 102, eachof the other cores 102 (i.e., not the core 102 at block 2602) getsinterrupted and receives the message at block 2652 and performs thesteps at blocks 2608 through 2634. Flow proceeds from block 2652 toblock 2608.

At block 2608, the core 102 writes a sync request to its sync register108 with a sync condition value of 21 (denoted sync 21 in FIG. 26), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 21. Flowproceeds to decision block 2611.

At decision block 2611, the core 102 determines whether it was the core102 that encountered the apply microcode patch instruction at block 2602(as opposed to a core 102 that received the patch message at block2652). If so, flow proceeds to block 2612; otherwise, flow proceeds toblock 2614.

At block 2612, the core 102 loads the immediate patch 2504 portion ofthe microcode patch 2500 into the uncore patch RAM 2408. Additionally,the core 102 generates a checksum of the loaded immediate patch 2504 andverifies that it matches the checksum 2506. Preferably, the core 102also sends information to the other cores 102 that specifies the lengthof the immediate patch 2504 and the location within the uncore patch RAM2408 to which the immediate patch 2504 was loaded. Advantageously,because all of the cores 102 are known to be executing the samemicrocode that implements the application of the microcode patch, if aprevious RAM patch 2516 is present in the uncore patch RAM 2408, it issafe to overwrite uncore patch RAM 2408 with the new patch because therewill be no hits in the patch CAM 2439 during this time (assuming themicrocode that implements the application of the microcode patch is notpatched). In an alternate embodiment, the core 102 loads the immediatepatch 2504 into the uncore PRAM 116, and prior to execution of theimmediate patch 2504 at block 2616, the core 102 copies the immediatepatch 2504 from the uncore PRAM 116 to the uncore patch RAM 2408.Preferably, the core 102 loads the immediate patch 2504 into a portionof the uncore PRAM 116 that is reserved for such purpose, e.g., aportion of the uncore PRAM 116 that is not being used for anotherpurpose, such as holding values used by the microcode (e.g., core 102state, TPM state, or effective microcode constants described above) andwhich may be patched (e.g., at block 2632), so that any previous uncorePRAM patch 2518 is not clobbered. In one embodiment, the loading intoand copying from the reserved portion of the uncore PRAM 116 areperformed in multiple stages in order to reduce the size required forthe reserved portion. Flow proceeds to block 2614.

At block 2614, the core 102 writes a sync request to its sync register108 with a sync condition value of 22 (denoted sync 22 in FIG. 26), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 22. Flowproceeds to block 2616.

At block 2616, the core 102 executes the immediate patch 2504 from theuncore patch RAM 2408. As described above, in one embodiment the core102 copies the immediate patch 2504 from the uncore PRAM 116 to theuncore patch RAM 2408 before executing the immediate patch 2504. Flowproceeds to block 2618.

At block 2618, the core 102 writes a sync request to its sync register108 with a sync condition value of 23 (denoted sync 23 in FIG. 26), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 23. Flowproceeds to decision block 2621.

At decision block 2621, the core 102 determines whether it was the core102 that encountered the apply microcode patch instruction at block 2602(as opposed to a core 102 that received the patch message at block2652). If so, flow proceeds to block 2622; otherwise, flow proceeds toblock 2624.

At block 2622, the core 102 loads the CAM data 2508 and core PRAM patch2512 into the uncore PRAM 116. Additionally, the core 102 generates achecksum of the loaded CAM data 2508 and core PRAM patch 2512 andverifies that it matches the checksum 2514. Preferably, the core 102also sends information to the other cores 102 that specifies the lengthof the CAM data 2508 and core PRAM patch 2512 and the location withinthe uncore PRAM 116 to which the CAM data 2508 and core PRAM patch 2512were loaded. Preferably, the core 102 loads the CAM data 2508 and corePRAM patch 2512 into a reserved portion of the uncore PRAM 116 so thatany previous uncore PRAM patch 2518 is not clobbered, similar to themanner described above with respect to block 2612. Flow proceeds toblock 2624.

At block 2624, the core 102 writes a sync request to its sync register108 with a sync condition value of 24 (denoted sync 24 in FIG. 26), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 24. Flowproceeds to block 2626.

At block 2626, the core 102 loads the CAM data 2508 from the uncore PRAM116 into its patch CAM 2439. Additionally, the core 102 loads the corePRAM patch 2512 from the uncore PRAM 116 into its core PRAM 2499.Advantageously, because all of the cores 102 are known to be executingthe same microcode that implements the application of the microcodepatch, even though the corresponding RAM patch 2516 has not yet beenwritten into the uncore patch RAM 2408 (which will occur at block 2632),it is safe to load the patch CAM 2439 with the new CAM data 2508 becausethere will be no hits in the patch CAM 2439 during this time (assumingthe microcode that implements the application of the microcode patch isnot patched). Additionally, any updates to the core PRAM 2499 by thecore PRAM patch 2512, including updates that change values that couldaffect the operation of the core 102 (e.g., feature set), are guaranteednot to be architecturally visible until the patch 2500 has beenpropagated to all the cores 102 because all of the cores 102 are knownto be executing the same microcode that implements the apply microcodepatch instruction and interrupts will not be enabled on any of the cores102 until the patch 2500 has been propagated to all the cores 102. Flowproceeds to block 2628.

At block 2628, the core 102 writes a sync request to its sync register108 with a sync condition value of 25 (denoted sync 25 in FIG. 26), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 25. Flowproceeds to decision block 2631.

At decision block 2631, the core 102 determines whether it was the core102 that encountered the apply microcode patch instruction at block 2602(as opposed to a core 102 that received the patch message at block2652). If so, flow proceeds to block 2632; otherwise, flow proceeds toblock 2634.

At block 2632, the core 102 loads the RAM patch 2516 into the uncorepatch RAM 2408. Additionally, the core 102 loads the uncore PRAM patch2518 into the uncore PRAM 116. In one embodiment, the uncore PRAM patch2518 includes code that is executed by the SPU 2423. In one embodiment,the uncore PRAM patch 2518 includes updates to values used by themicrocode, as described above. In one embodiment, the uncore PRAM patch2518 includes both the SPU 2423 code and updates to values used by themicrocode. Advantageously, it is safe to load the RAM patch 2516 intothe uncore patch RAM 2408 because all of the cores 102 are known to beexecuting the same microcode that implements the application of themicrocode patch, more specifically, that the patch CAM 2439 of all thecores 102 has already been loaded with the new CAM data 2508 (e.g., atblock 2626), and there will be no hits in the patch CAM 2439 during thistime (assuming the microcode that implements the application of themicrocode patch is not patched). Additionally, any updates to the uncorePRAM 116 by the uncore PRAM patch 2518, including updates that changevalues that could affect the operation of the core 102 (e.g., featureset), are guaranteed not to be architecturally visible until the patch2500 has been propagated to all the cores 102 because all of the cores102 are known to be executing the same microcode that implements theapply microcode patch instruction and interrupts will not be enabled onany of the cores 102 until the patch 2500 has been propagated to all thecores 102. Flow proceeds to block 234.

At block 2634, the core 102 writes a sync request to its sync register108 with a sync condition value of 26 (denoted sync 26 in FIG. 26), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 26. Flow ends atblock 2634.

After block 2634, if code was loaded into the uncore PRAM 116 for theSPU 2423 to execute at block 2632, the patch core 102 also subsequentlycauses the SPU 2423 to begin executing the code, as described below withrespect to FIG. 30. Also, after block 2634, the patch core 102 releasesthe hardware semaphore 118 obtained at block 2604. Still further, afterblock 2634, the core 102 re-enables interrupts.

Referring now to FIG. 27, a timing diagram illustrating an example ofthe operation of a microprocessor according to the flowchart of FIG. 26is shown. In the example, a configuration of a microprocessor 100 withthree cores 102, denoted core 0, core 1 and core 2, is shown; however,it should be understood that in other embodiments the microprocessor 100may include different numbers of cores 102. In the timing diagram, thetiming of events proceeds downward.

Core 0 receives a request to patch microcode (per block 2602) and inresponse obtains the hardware semaphore 118 (per block 2604). Core 0then sends a microcode patch message and interrupt to core 1 and core 2(per block 2606). Core 0 then writes a sync 21 and is put to sleep (perblock 2608).

Each of core 1 and core 2 eventually are interrupted from their currenttasks and read the message (per block 2652). In response, each of core 1and core 2 writes a sync 21 and is put to sleep (per block 2608). Asshown, the time at which each of the cores writes the sync 21 may vary,for example due to the latency of the instruction that is executing whenthe interrupt is asserted.

When all the cores have written the sync 21, the control unit 104 wakesthem all up simultaneously (per block 2608). Core 0 then loads theimmediate patch 2504 into uncore PRAM 116 (per block 2612) and writes async 22 and is put to sleep (per block 2614). Core 1 and core 2 eachwrite a sync 22 and is put to sleep (per block 2614).

When all the cores have written the sync 22, the control unit 104 wakesthem all up simultaneously (per block 2614). Each core then executes theimmediate patch 2504 (per block 2616) and writes a sync 23 and is put tosleep (per block 2618).

When all the cores have written the sync 23, the control unit 104 wakesthem all up simultaneously (per block 2618). Core 0 then loads the CAMdata 2508 and core PRAM patch 2512 into uncore PRAM 116 (per block 2622)and writes a sync 24 and is put to sleep (per block 2624). Core 1 andcore 2 each write a sync 24 and is put to sleep (per block 2624).

When all the cores have written the sync 24, the control unit 104 wakesthem all up simultaneously (per block 2624). Each core then loads itspatch CAM 2439 with the CAM data 2508 and loads its core PRAM 2499 withthe core PRAM patch 2512 (per block 2626) and writes a sync 25 and isput to sleep (per block 2628).

When all the cores have written the sync 25, the control unit 104 wakesthem all up simultaneously (per block 2628). Core 0 then loads the RAMpatch 2516 into the uncore microcode patch RAM 2408 and loads the uncorePRAM patch 2518 into uncore PRAM 116 (per block 2632) and writes a sync26 and is put to sleep (per block 2634). Core 1 and core 2 each write async 26 and is put to sleep (per block 2634).

When all the cores have written the sync 26, the control unit 104 wakesthem all up simultaneously (per block 2634). As described above, if codewas loaded into the uncore PRAM 116 for the SPU 2423 to execute at block2632, the core 102 also subsequently causes the SPU 2423 to beginexecuting the code, as described below with respect to FIG. 30.

Referring now to FIG. 28, a block diagram illustrating a multicoremicroprocessor 100 according to an alternate embodiment is shown. Themicroprocessor 100 is similar in many respects to the microprocessor 100of FIG. 24. However, the microprocessor 100 of FIG. 28 does not includean uncore patch RAM, but instead includes a core patch RAM 2808 in eachof the cores 102, which serves a similar function to the uncore patchRAM 2408 of FIG. 24; however, the core patch RAM 2808 in each of thecores 102 is private to its respective core 102 and is not shared by theother cores 102.

Referring now to FIG. 29, a flowchart illustrating operation of themicroprocessor 100 of FIG. 28 to propagate a microcode patch to multiplecores 102 of the microprocessor 100 according to an alternate embodimentis shown. In the alternate embodiment of FIGS. 28 and 29, the patch 2500of FIG. 25 may be modified such that the checksum 2514 follows the RAMpatch 2516, rather than the core PRAM patch 2512, and enables themicroprocessor 100 to verify the integrity of the CAM data 2508, corePRAM patch 2512 and RAM patch 2516 after they are loaded into themicroprocessor 100 (e.g., at block 2922 of FIG. 29). The flowchart ofFIG. 29 is similar in many respects to the flowchart of FIG. 26 andsimilarly numbered blocks are similar. However, block 2912 replacesblock 2612, block 2916 replaces block 2616, block 2922 replaces block2622, block 2926 replaces block 2626, and block 2932 replaces block2632. At block 2912, the core 102 loads the immediate patch 2504 intothe uncore PRAM 116 (rather than into an uncore patch RAM). At block2916, the core 102 copies the immediate patch 2504 from the uncore PRAM116 to the core patch RAM 2808 before executing it. At block 2922, thecore 102 loads the RAM patch 2516, in addition to the CAM data 2508 andcore PRAM patch 2512, into the uncore PRAM 116. At block 2926, the core102 loads the RAM patch 2516 from the uncore PRAM 116 into its patch RAM2808, in addition to loading the CAM data 2508 from the uncore PRAM 116into its patch CAM 2439 and loading the core PRAM patch 2512 from theuncore PRAM 116 into its core PRAM 2499. At block 2932, unlike at block2632 of FIG. 26, the core 102 does not load the RAM patch 2516 into anuncore patch RAM.

As may be observed from the above embodiment, advantageously the atomicpropagation of the microcode patch 2500 to each of relevant memories2439/2499/2808 of the cores 102 of the microprocessor 100 and to therelevant uncore memories 2408/116 is performed in a manner to insure theintegrity and efficacy of the patch 2500 even in the presence ofmultiple concurrently executing cores 102 that share resources and thatmight otherwise clobber various portions of one another's patches ifapplied in the conventional manner.

Patching Service Processor Code

Referring now to FIG. 30, a flowchart illustrating operation of themicroprocessor 100 of FIG. 24 to patch code for a service processor isshown. Flow begins at block 3002.

At block 3002, the core 102 loads code to be executed by the SPU 2423into the uncore PRAM 116 at a patch address specified by the patch, suchas described above with respect to block 2632 of FIG. 26. Flow proceedsto block 3004.

At block 3004, the core 102 controls the SPU 2423 to execute code at thepatch address, i.e., the address in uncore PRAM 116 to which the SPU2423 code was written at block 3002. In one embodiment, the SPU 2423 isconfigured to fetch its reset vector (i.e., the address at which the SPU2423 begins to fetch instructions after coming out of reset) from thestart address register 2497, and the core 102 writes the patch addressinto the start address register 2497 and then writes to a controlregister that causes the SPU 2423 to be reset. Flow proceeds to block3006.

At block 3006, the SPU 2423 begins fetching code (i.e., fetches itsfirst instruction) at the patch address, i.e., at the address in uncorePRAM 116 to which the SPU 2423 code was written at block 3002.Typically, the SPU 2423 patch code residing in the uncore PRAM 116 willperform a jump to SPU 2423 code residing in the uncore microcode ROM2425. Flow ends at block 3006.

The ability to patch the SPU 2423 code may be particularly useful. Forexample, the SPU 2423 may be used for performance testing that istransient in nature, i.e., it may not desirable to make the performancetesting SPU 2423 code a permanent part of the microprocessor 100, e.g.s,for production parts, but rather only part of development parts. Foranother example, the SPU 2423 may be used to find and/or fix bugs. Foranother example, the SPU 2423 may be used to configure themicroprocessor 100.

Atomic Propagation of Updates to Per-Core-InstantiatedArchitecturally-Visible Storage Resources

Referring now to FIG. 31, a block diagram illustrating a multicoremicroprocessor 100 according to an alternate embodiment is shown. Themicroprocessor 100 is similar in many respects to the microprocessor 100of FIG. 24. However, each core 102 of the microprocessor 100 of FIG. 31also includes architecturally-visible memory type range registers(MTRRs) 3102. That is, each core 102 instantiates thearchitecturally-visible MTRRs 3102, even though system software expectsthe MTRRs 3102 to be consistent across all the cores 102 (as describedin more detail below). The MTRRs 3102 are examples ofper-core-instantiated architecturally-visible storage resources, andother embodiments of per-core-instantiated architecturally-visiblestorage resources are described below. (Although not shown, each core102 also includes the core PRAM 2499, core microcode ROM 2404, patch CAM2439 of FIG. 24 and, in one embodiment, the core microcode patch RAM2808 of FIG. 28.)

The MTRRs 3102 provide a way for system software to associate a memorytype with multiple different physical address ranges in the systemmemory address space of the microprocessor 100. Examples of differentmemory types include strong uncacheable, uncacheable, write-combining,write through, write back and write protected. Each MTRR 3102 specifiesa memory range (either explicitly or implicitly) and its memory type.The collective values of the various MTRRs 3102 define a memory map thatspecifies the memory type of the different memory ranges. In oneembodiment, the MTRRs 3102 are similar to the description in the Intel64 and IA-32 Architectures Software Developer's Manual, Volume 3: SystemProgramming Guide, September 2013, particularly in section 11.11, whichis hereby incorporated by reference in its entirety for all purposes.

It is desirable that the memory map defined by the MTRRs 3102 beidentical for all the cores 102 of the microprocessor 100 so thatsoftware running on the microprocessor 100 has a consistent view ofmemory. However, in a conventional processor, there is no hardwaresupport for maintaining consistency of the MTRRs between the cores of amulti-core processor. As stated in the NOTE at the bottom of page 11-20of Volume 3 of the above-referenced Intel Manual, “The P6 and morerecent processor families provide no hardware support for maintainingthis consistency [of MTRR values].” Consequently, system software isresponsible for maintaining the MTRR consistency across cores. Section11.11.8 of the above-referenced Intel Manual describes an algorithm forsystem software to maintain the consistency that involves each core ofthe multi-core processor updating its MTRRs, i.e., all of the coresexecute instructions to update their respective MTRRs.

In contrast, embodiments are described herein in which the systemsoftware may update the respective instance of the MTRR 3102 on one ofthe cores 102, and that core 102 advantageously propagates the update tothe respective instance of the MTRR 3102 on all of the cores 102 of themicroprocessor 100 in an atomic fashion (somewhat similar to the mannerin which a microcode patch is performed as described above with respectto the embodiments of FIGS. 24 through 30). This provides a means formaintaining consistency at the architectural instruction level betweenthe MTRRs 3102 of the different cores 102.

Referring now to FIG. 32, a flowchart illustrating operation of themicroprocessor 100 of FIG. 31 to propagate an MTRR 3102 update tomultiple cores 102 of the microprocessor 100 is shown. The operation isdescribed from the perspective of a single core, but each of the cores102 of the microprocessor 100 operates according to the description tocollectively propagate the MTRR 3102 update to all of the cores 102 ofthe microprocessor 100. More specifically, FIG. 32 describes theoperation of one core that encounters the instruction to update the MTRR3102, whose flow begins at block 3202, and the operation of the othercores 102, whose flow begins at block 3252.

At block 3202, one of the cores 102 encounters an architectural xinstruction that instructs the core 102 to update an MTRR 3102 of thecore 102. That is, the MTRR update instruction includes an MTRR 3102identifier and an update value to be written to the MTRR 3102. In oneembodiment, the MTRR update instruction is an x86 WRMSR instruction thatspecifies the update value in the EAX:EDX registers and the MTRR 3102identifier in the ECX register, which is an MSR address within the MSRaddress space of the core 102. In response to the MTRR updateinstruction, the core 102 disables interrupts and traps to microcodethat implements the MTRR update instruction. It should be understandthat the system software that includes the MTRR update instruction mayinclude a sequence of multiple instructions to prepare for the update ofthe MTRR 3102; however, preferably, it is in response to a singlearchitectural instruction of the sequence that the MTRR 3102 all of thecores 102 is updated in an atomic fashion at the architecturalinstruction level. That is, once interrupts are disabled on the firstcore 102 (i.e., the core 102 that encounters the MTRR update instructionat block 3202), interrupts remain disabled while the implementingmicrocode propagates the new MTRR 3102 value to all the cores 102 of themicroprocessor 100 (e.g., until after block 3218); furthermore, onceinterrupts are disabled on the other cores 102 (e.g., at block 3252),they remain disabled until the MTRR 3102 of all the cores 102 of themicroprocessor 100 have been updated (e.g., until after block 3218).Thus, advantageously, the new MTRR 3102 value is propagated to all ofthe cores 102 of the microprocessor 100 in an atomic fashion at thearchitectural instruction level. Flow proceeds to block 3204.

At block 3204, the core 102 obtains ownership of the hardware semaphore118 of FIG. 1. Preferably, the microprocessor 100 includes a hardwaresemaphore 118 associated with an MTRR 3102 update. Preferably, the core102 obtains ownership of the hardware semaphore 118 in a manner similarto that described above with respect to FIG. 20, and more particularlywith respect to block 2004 and 2006. The hardware semaphore 118 is usedbecause it is possible while one of the cores 102 is performing an MTRR3102 update in response to encountering an MTRR update instruction, asecond core 102 encounters an MTRR update instruction, in response towhich the second core would begin to update the MTRR 3102, which mightresult in incorrect execution. Flow proceeds to block 3206.

At block 3206, the core 102 sends a MTRR update message to the othercores 102 and sends them an inter-core interrupt. Preferably, the core102 traps to microcode in response to the MTRR update instruction (atblock 3202) or in response to the interrupt (at block 3252) and remainsin microcode, during which time interrupts are disabled (i.e., themicrocode does not allow itself to be interrupted), until block 3218.Flow proceeds from block 3206 to block 3208.

At block 3252, one of the other cores 102 (i.e., a core 102 other thanthe core 102 that encountered the MTRR update instruction at block 3202)gets interrupted and receives the MTRR update message as a result of theinter-core interrupt sent at block 3206. In one embodiment, the core 102takes the interrupt at the next architectural instruction boundary(e.g., at the next x86 instruction boundary). In response to theinterrupt, the core 102 disables interrupts and traps to microcode thathandles the MTRR update message. As described above, although flow atblock 3252 is described from the perspective of a single core 102, eachof the other cores 102 (i.e., not the core 102 at block 3202) getsinterrupted and receives the message at block 3252 and performs thesteps at blocks 3208 through 3234. Flow proceeds from block 3252 toblock 3208.

At block 3208, the core 102 writes a sync request to its sync register108 with a sync condition value of 31 (denoted sync 31 in FIG. 32), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 31. Flowproceeds to decision block 3211.

At decision block 3211, the core 102 determines whether it was the core102 that encountered the MTRR update instruction at block 3202 (asopposed to a core 102 that received the MTRR update message at block3252). If so, flow proceeds to block 3212; otherwise, flow proceeds toblock 3214.

At block 3212, the core 102 loads into the uncore PRAM 116 the MTRRidentifier specified by the MTRR update instruction and an MTRR updatevalue with which the MTRR is to be updated such that it is visible byall the other cores 102. In the case of an x86 embodiment, MTRRs 3102include both (1) fixed range MTRRs that comprise a single 64-bit MSRthat is updated via a single WRMSR instruction and (2) variable rangeMTRRs that comprise two 64-bit MSRs each of which is written via adifferent WRMSR instruction, i.e., the two WRMSR instructions specifydifferent MSR addresses. For variable range MTRRs, one of the MSRs (thePHYSBASE register) includes a base address of the memory range and atype field for specifying the memory type, and the other of the MSRs(the PHYSMASK register) includes a valid bit and a mask field that setsthe range mask. Preferably, the core 102 loads into the uncore PRAM 116the MTRR update value as follows.

-   -   1. If the MSR identified is the PHYSMASK register, the core 102        loads into the uncore PRAM 116 a 128-bit update value that        includes both the new 64-bit value specified by the WRMSR        instruction (which includes the valid bit and mask values) and        the current value of the PHYSBASE register (which includes the        base and type values).    -   2. If the MSR identified is the PHYSBASE register:        -   a. If the valid bit in the PHYSMASK register is currently            set, the core 102 loads into the uncore PRAM 116 a 128-bit            update value that includes both the new 64-bit value            specified by the WRMSR instruction (which includes the base            and type values) and the current value of the PHYSMASK            register (which includes the valid bit and mask values).        -   b. If the valid bit in the PHYSMASK register is currently            clear, the core 102 loads into the uncore PRAM 116 a 64-bit            update value that includes only the new 64-bit value            specified by the WRMSR instruction (which includes the base            and type values).            Additionally, the core 102 sets a flag in the uncore PRAM            116 if the update value written is a 128-bit value and            clears the flag if the update value is a 64-bit value. Flow            proceeds from block 3212 to block 3214.

At block 3214, the core 102 writes a sync request to its sync register108 with a sync condition value of 32 (denoted sync 32 in FIG. 32), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 32. Flowproceeds to block 3216.

At block 3216, the core 102 reads from the uncore PRAM 116 the MTRR 3102identifier and MTRR update value written at block 3212 and updates itsidentified MTRR 3102 with the MTRR update value. Advantageously, theMTRR update value propagation is performed in an atomic fashion suchthat any updates to the MTRRs 3102 that could affect the operation oftheir respective core 102 are guaranteed not to be architecturallyvisible until the update value has been propagated to the MTRR 3102 ofall the cores 102 because all of the cores 102 are known to be executingthe same microcode that implements the MTRR update instruction andinterrupts will not be enabled on any of the cores 102 until the valuehas been propagated to the respective MTRR 3102 of all the cores 102.With respect to the embodiment described above with respect to block3212, if the flag written at block 3212 is set, the core 102 alsoupdates (in addition to the identified MSR) the PHYSMASK or PHYSBASEregister; otherwise if the flag is clear, the core 102 only updates theidentified MSR. Flow proceeds to block 3218.

At block 3218, the core 102 writes a sync request to its sync register108 with a sync condition value of 33 (denoted sync 33 in FIG. 32), isput to sleep by the control unit 104, and subsequently awakened by thecontrol unit 104 when all cores 102 have written a sync 33. Flow ends atblock 3218.

After block 3218, the MTRR core 102 releases the hardware semaphore 118obtained at block 3204. Still further, after block 3218, the core 102re-enables interrupts.

As may be observed from FIGS. 31 and 32, system software running on themicroprocessor 100 of FIG. 31 may advantageously execute a MTRR updateinstruction on a single core 102 of the microprocessor 100 to accomplishupdating of the specified MTRR 3102 of all the cores 102 of themicroprocessor 100, rather than executing a MTRR update instruction oneach of the cores 102 individually, which may provide system integrityadvantages.

One particular MTRR 3102 instantiated in each core 102 is a systemmanagement range register (SMRR) 3102. The memory range specified by theSMRR 3102 is referred to as the SMRAM region because it holds code anddata associated with system management mode (SMM) operation, such as asystem management interrupt (SMI) handler. When code running on a core102 attempts to access the SMRAM region, the core 102 only allows theaccess if the core 102 is running in SMM; otherwise, the core 102ignores a write to the SMRAM region and returns a fixed value for eachbyte read from the SMRAM region. Furthermore, if a core 102 running inSMM attempts to execute code outside the SMRAM region, the core 102 willassert a machine check exception. Additionally, the core 102 only allowscode to write the SMRR 3102 if it is running in SMM. This facilitatesthe protection of SMM code and data in the SMRAM region. In oneembodiment, the SMRR 3102 is similar to that described in the Intel 64and IA-32 Architectures Software Developer's Manual, Volume 3: SystemProgramming Guide, September 2013, particularly in sections 11.11.2.4and 34.4.2.1, which are hereby incorporated by reference in theirentirety for all purposes.

Typically, each core 102 has its own instance of SMM code and data inmemory. It is desirable that each core's 102 SMM code and data beprotected not only from code running on itself, but also from coderunning on the other cores 102. To accomplish this using the SMRRs 3102,system software typically places the multiple SMM code and datainstances in adjacent blocks of memory. That is, the SMRAM region is asingle contiguous memory region that includes all of the SMM code anddata instances. If the SMRR 3102 of all the cores 102 of themicroprocessor 100 have values that specify the entirety of the singlecontiguous memory region that includes all of the SMM code and datainstances, this prevents code running on one core in non-SMM fromupdating the SMM code and data instance of another core 102. If a windowof time exists in which the SMRR 3102 values of the cores 102 aredifferent, i.e., the SMRRs 3102 of different cores 102 of themicroprocessor 100 have different values any of which specify less thanthe entirety of the single contiguous memory region that includes all ofthe SMM code and data instances, then the system may be vulnerable to asecurity attack, which may be serious given the nature of SMM. Thus,embodiments that atomically propagate updates to the SMRRs 3102 may beparticularly advantageous.

Additionally, other embodiments are contemplated in which the update ofother per-core-instantiated architecturally-visible storage resources ofthe microprocessor 100 are propagated in an atomic fashion similar tothe manner described above. For example, in one embodiment each core 102instantiates certain bit fields of the x86 IA32_MISC_ENABLE MSR, and aWRMSR executed on one core 102 is propagated to all of the cores 102 ofthe microprocessor 100 in a manner similar to that described above.Furthermore, embodiments are contemplated in which the execution on onecore 102 of a WRMSR to other MSRs that are instantiated on all of thecores 102 of the microprocessor 100, both architectural and proprietaryand/or current and future, is propagated to all of the cores 102 of themicroprocessor 100 in a manner similar to that described above.

Furthermore, although embodiments are described in which theper-core-instantiated architecturally-visible storage resources areMTRRs, other embodiments are contemplated in which theper-core-instantiated resources are resources of different instructionset architectures than the x86 ISA, and are other resources than MTRRs.For example, other resources than the MTRRs include CPUID values andMSRs that report capabilities, such as Vectored Multimedia eXtensions(VMX) capabilities.

While various embodiments of the present invention have been describedherein, it should be understood that they have been presented by way ofexample, and not limitation. It will be apparent to persons skilled inthe relevant computer arts that various changes in form and detail canbe made therein without departing from the scope of the invention. Forexample, software can enable, for example, the function, fabrication,modeling, simulation, description and/or testing of the apparatus andmethods described herein. This can be accomplished through the use ofgeneral programming languages (e.g., C, C++), hardware descriptionlanguages (HDL) including Verilog HDL, VHDL, and so on, or otheravailable programs. Such software can be disposed in any known computerusable medium such as magnetic tape, semiconductor, magnetic disk, oroptical disc (e.g., CD-ROM, DVD-ROM, etc.), a network, wire line,wireless or other communications medium. Embodiments of the apparatusand method described herein may be included in a semiconductorintellectual property core, such as a microprocessor core (e.g.,embodied, or specified, in a HDL) and transformed to hardware in theproduction of integrated circuits. Additionally, the apparatus andmethods described herein may be embodied as a combination of hardwareand software. Thus, the present invention should not be limited by anyof the exemplary embodiments described herein, but should be definedonly in accordance with the following claims and their equivalents.Specifically, the present invention may be implemented within amicroprocessor device that may be used in a general-purpose computer.Finally, those skilled in the art should appreciate that they canreadily use the disclosed conception and specific embodiments as a basisfor designing or modifying other structures for carrying out the samepurposes of the present invention without departing from the scope ofthe invention as defined by the appended claims.

1. A microprocessor, comprising: a plurality of processing cores; anuncore random access memory (RAM), readable and writable by each of theplurality of processing cores; wherein each core of the plurality ofprocessing cores comprises microcode run by the core that implementsarchitectural instructions of an instruction set architecture of themicroprocessor; and wherein the microcode is configured to both read andwrite the uncore RAM to accomplish inter-core communication between theplurality of processing cores.
 2. The microprocessor of claim 1, whereinthe uncore RAM is not in the architectural user program address space ofthe microprocessor.
 3. The microprocessor of claim 1, wherein themicrocode of each core of the plurality of processing cores isconfigured to implement a trusted platform module (TPM), wherein themicrocode is configured such that only one of the plurality ofprocessing cores implements the TPM at each instance of time, whereinthe inter-core communication between the plurality of processing corescomprises a transfer of state of the TPM from a first of the pluralityof processing cores to a second of the plurality of processing cores. 4.The microprocessor of claim 1, wherein the inter-core communicationbetween the plurality of processing cores comprises: a first of theplurality of processing cores, in response to encountering anarchitectural instruction that instructs the microprocessor toinvalidate its cache memories, sends a message via the uncore RAM toeach of the other plurality of processing cores to invalidate core-localcache memories.
 5. The microprocessor of claim 1, wherein the inter-corecommunication between the plurality of processing cores comprises: afirst of the plurality of processing cores sends a message via theuncore RAM to each of the other plurality of processing cores to dumpits state for debugging purposes.
 6. The microprocessor of claim 1,wherein the inter-core communication between the plurality of processingcores comprises: a first of the plurality of processing cores sends amessage via the uncore RAM to each of the other plurality of processingcores to apply a microcode patch.
 7. The microprocessor of claim 1,wherein the inter-core communication between the plurality of processingcores further comprises: a first of the plurality of processing coreswrites a microcode patch to the uncore RAM and each of the otherplurality of processing cores reads the microcode patch from the uncoreRAM.
 8. The microprocessor of claim 1, wherein the inter-corecommunication between the plurality of processing cores comprises: afirst of the plurality of processing cores, in response to encounteringan architectural instruction that instructs the first core to disableitself, sends a message via the uncore RAM to each of the otherplurality of processing cores to perform a query to determine which ofthe plurality of processing cores of the microprocessor are enabled. 9.The microprocessor of claim 1, wherein the inter-core communicationbetween the plurality of processing cores further comprises: a first ofthe plurality of processing cores writes a value to the uncore RAM andeach of the other plurality of processing cores reads the value from theuncore RAM; and wherein each of the plurality of processing cores isconfigured to update a respective per-core-instantiatedarchitecturally-visible storage resource with the value.
 10. Themicroprocessor of claim 1, wherein the inter-core communication betweenthe plurality of processing cores further comprises: a first of theplurality of processing cores writes a value to the uncore RAMindicating a number of processing cores of the microprocessor; andwherein each of the plurality of processing cores is configured to readthe value from the uncore RAM to perform an architectural instructionthat requests the number of processing cores of the microprocessor. 11.A method to be performed by a microprocessor having a plurality ofprocessing cores and an uncore random access memory (RAM) readable andwritable by each of the plurality of processing cores, wherein each coreof the plurality of processing cores comprises microcode run by the corethat implements architectural instructions of an instruction setarchitecture of the microprocessor, the method comprising: reading, bythe microcode, to the uncore RAM; writing, by the microcode, from theuncore RAM; wherein said reading and writing accomplish inter-corecommunication between the plurality of processing cores.
 12. The methodof claim 11, wherein the uncore RAM is not in the architectural userprogram address space of the microprocessor.
 13. The method of claim 11,wherein the microcode of each core of the plurality of processing coresis configured to implement a trusted platform module (TPM), wherein themicrocode is configured such that only one of the plurality ofprocessing cores implements the TPM at each instance of time, whereinthe inter-core communication between the plurality of processing corescomprises a transfer of state of the TPM from a first of the pluralityof processing cores to a second of the plurality of processing cores.14. The method of claim 11, wherein the inter-core communication betweenthe plurality of processing cores comprises: a first of the plurality ofprocessing cores, in response to encountering an architecturalinstruction that instructs the microprocessor to invalidate its cachememories, sending a message via the uncore RAM to each of the otherplurality of processing cores to invalidate core-local cache memories.15. The method of claim 11, wherein the inter-core communication betweenthe plurality of processing cores comprises: a first of the plurality ofprocessing cores sending a message via the uncore RAM to each of theother plurality of processing cores to dump its state for debuggingpurposes.
 16. The method of claim 11, wherein the inter-corecommunication between the plurality of processing cores comprises: afirst of the plurality of processing cores sending a message via theuncore RAM to each of the other plurality of processing cores to apply amicrocode patch.
 17. The method of claim 11, wherein the inter-corecommunication between the plurality of processing cores furthercomprises: a first of the plurality of processing cores writing amicrocode patch to the uncore RAM and each of the other plurality ofprocessing cores reading the microcode patch from the uncore RAM. 18.The method of claim 11, wherein the inter-core communication between theplurality of processing cores comprises: a first of the plurality ofprocessing cores, in response to encountering an architecturalinstruction that instructs the first core to disable itself, sending amessage via the uncore RAM to each of the other plurality of processingcores to perform a query to determine which of the plurality ofprocessing cores of the microprocessor are enabled.
 19. The method ofclaim 11, wherein the inter-core communication between the plurality ofprocessing cores further comprises: a first of the plurality ofprocessing cores writing a value to the uncore RAM and each of the otherplurality of processing cores reading the value from the uncore RAM; andupdating, by each of the plurality of processing cores, a respectiveper-core-instantiated architecturally-visible storage resource with thevalue.
 20. The method of claim 11, wherein the inter-core communicationbetween the plurality of processing cores further comprises: a first ofthe plurality of processing cores writing a value to the uncore RAMindicating a number of processing cores of the microprocessor; andreading, by each of the plurality of processing cores, the value fromthe uncore RAM to perform an architectural instruction that requests thenumber of processing cores of the microprocessor.
 21. A computer programproduct encoded in at least one non-transitory computer usable mediumfor use with a computing device, the computer program productcomprising: computer usable program code embodied in said medium, forspecifying a microprocessor, the computer usable program codecomprising: first program code for specifying a plurality of processingcores; and second program code for specifying an uncore random accessmemory (RAM), readable and writable by each of the plurality ofprocessing cores; wherein each core of the plurality of processing corescomprises microcode run by the core that implements architecturalinstructions of an instruction set architecture of the microprocessor;and wherein the microcode is configured to both read and write theuncore RAM to accomplish inter-core communication between the pluralityof processing cores.